Deep learning medical systems and methods for medical procedures

ABSTRACT

Methods and apparatus for monitoring and improving imaging system operation are provided. An example apparatus includes a first deployed deep learning network (DLN) which operates with an acquisition engine to generate an imaging device configuration. The example apparatus includes a second deployed DLN which operates with a reconstruction engine based on acquired image data. The example apparatus includes a first assessment engine with a third deployed DLN. The assessment engine receives output from at least one of the acquisition engine or the reconstruction engine to assess operation of the respective at least one of the acquisition engine or the reconstruction engine and to provide feedback to the respective at least one of the acquisition engine or the reconstruction engine. The first deployed DLN and the second deployed DLN are generated and deployed from first and second training DLNS, respectively.

FIELD OF THE DISCLOSURE

This disclosure relates generally to improved medical systems and, moreparticularly, to improved deep learning medical systems and methods formedical procedures.

BACKGROUND

A variety of economy, technological, and administrative hurdleschallenge healthcare facilities, such as hospitals, clinics, doctors'offices, etc., to provide quality care to patients. Economic drivers,less skilled staff, fewer staff, complicated equipment, and emergingaccreditation for controlling and standardizing radiation exposure doseusage across a healthcare enterprise create difficulties for effectivemanagement and use of imaging and information systems for examination,diagnosis, and treatment of patients.

Healthcare provider consolidations create geographically distributedhospital networks in which physical contact with systems is too costly.At the same time, referring physicians want more direct access tosupporting data in reports along with better channels for collaboration.Physicians have more patients, less time, and are inundated with hugeamounts of data, and they are eager for assistance.

BRIEF SUMMARY

Certain examples provide an apparatus including a first deployed deeplearning network associated with an acquisition engine, the acquisitionengine associated with an imaging device, the first deployed deeplearning network configured to operate with the acquisition engine togenerate a configuration for the imaging device, the first deployed deeplearning network generated and deployed from a first training deeplearning network. The example apparatus also includes a second deployeddeep learning network associated with a reconstruction engine, thereconstruction engine to receive acquired image data from the imagingdevice via the acquisition engine and to reconstruct an image from theacquired image data, the second deployed deep learning network tooperate with the reconstruction engine based on the acquired image data,the second deployed deep learning network generated and deployed from asecond training deep learning network. The example apparatus furtherincludes a first assessment engine with a third deployed deep learningnetwork, the assessment engine to receive output from at least one ofthe acquisition engine or the reconstruction engine to assess operationof the respective at least one of the acquisition engine or thereconstruction engine and to provide feedback to the respective at leastone of the acquisition engine or the reconstruction engine.

Certain examples provide a method including generating a configurationfor the imaging device for image acquisition via a first deployed deeplearning network associated with an acquisition engine associated withthe imaging device, the first deployed deep learning network generatedand deployed from a first training deep learning network. The examplemethod includes monitoring, using a second deployed deep learningnetwork, image reconstruction by a reconstruction engine of image dataacquired by the imaging device via the acquisition engine, the seconddeployed deep learning network associated with the reconstruction engineand to operate with the reconstruction engine based on the acquiredimage data, the second deployed deep learning network generated anddeployed from a second training deep learning network. The examplemethod includes assessing operation of respective at least one of theacquisition engine or the reconstruction engine based on output receivedfrom the respective at least one of the acquisition engine or thereconstruction engine. The example method includes providing feedback tothe respective at least one of the acquisition engine or thereconstruction engine.

Certain examples provide a computer readable medium includinginstructions. When executed, the example instructions cause a machine toat least generate a configuration for the imaging device for imageacquisition via a first deployed deep learning network associated withan acquisition engine associated with the imaging device, the firstdeployed deep learning network generated and deployed from a firsttraining deep learning network. When executed, the example instructionscause the machine to at least monitor, using a second deployed deeplearning network, image reconstruction by a reconstruction engine ofimage data acquired by the imaging device via the acquisition engine,the second deployed deep learning network associated with thereconstruction engine and to operate with the reconstruction enginebased on the acquired image data, the second deployed deep learningnetwork generated and deployed from a second training deep learningnetwork. When executed, the example instructions cause the machine to atleast assess operation of respective at least one of the acquisitionengine or the reconstruction engine, or the diagnosis engine based onoutput received from the respective at least one of the acquisitionengine or the reconstruction engine. When executed, the exampleinstructions cause the machine to at least provide feedback to therespective at least one of the acquisition engine or the reconstructionengine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a representation of an example deep learning neural network.

FIG. 2 illustrates a particular implementation of the example neuralnetwork as a convolutional neural network.

FIG. 3 is a representation of an example implementation of an imageanalysis convolutional neural network.

FIG. 4A illustrates an example configuration to apply a deep learningnetwork to process and/or otherwise evaluate an image.

FIG. 4B illustrates a combination of a plurality of deep learningnetworks.

FIG. 5 illustrates example training and deployment phases of a deeplearning network.

FIG. 6 illustrates an example product leveraging a trained networkpackage to provide a deep learning product offering.

FIGS. 7A-7C illustrate various deep learning device configurations.

FIGS. 8A-8B illustrate example learning and improvement factoriesleveraging deep learning networks.

FIG. 8C illustrates an example flow diagram of an example method totrain and deploy a deep learning network model.

FIG. 8D illustrates an example process to collect and store feedbackduring operation of a deployed deep learning network model-based deviceand re-train the model for re-deployment.

FIG. 9 illustrates an example system including a data factory,application factory, and learning factory leveraging deep learning toprovide applications for one or more systems and/or associated users.

FIG. 10 illustrates an overview of a medical device ecosystem includingdevices physically deployed internally and externally (physical factory)with a digital factory.

FIG. 11 illustrates an example physical device and its data flowinteracting with the digital factory.

FIG. 12 illustrates a flow diagram of an example method to process andleverage data in a digital factory.

FIG. 13 provides further detail regarding the example method to processand leverage data in the data factory and learning factory.

FIG. 14 illustrates an example healthcare system for patient evaluationand diagnosis with deep learning.

FIG. 15A illustrates a further detailed view of an example improvedhealthcare system for patient evaluation and diagnosis.

FIG. 15B illustrates an example system implementation in which theacquisition engine, reconstruction engine, and diagnosis engine areaccompanied by a data quality assessment engine, an image qualityassessment engine, and a diagnosis assessment engine.

FIG. 16 illustrates a flow diagram of an example method for improvedimage acquisition, processing, and patient diagnosis.

FIG. 17 illustrates an example data flow and transformation ofinformation as it flows among the components of the example system ofFIG. 15A.

FIG. 18 illustrates an example healthcare analytics framework for imageacquisition, image reconstruction, image analysis, and patient diagnosisusing the example systems of FIGS. 14-15B.

FIG. 19 illustrates a flow diagram of an example method for imageacquisition.

FIG. 20 illustrates example image acquisition configuration system.

FIG. 21 illustrates a flow diagram of an example method to train anddeploy the image acquisition configuration device of FIG. 20.

FIG. 22 shows a graph of an image quality index as a function of doseprovided to a patient.

FIGS. 23A-23B illustrate example learning and testing/evaluation phasesfor an image quality deep learning network.

FIGS. 24A-24B show example learning, validation, and testing phases foran example deep convolution network.

FIG. 25A shows an example trained network leveraged to determine anoutput quality for an initial set of reconstruction parameters.

FIG. 25B illustrates an example system for image quality assessment andfeedback using a deployed network model.

FIG. 25C illustrates an example system for detection and/or diagnosisassessment and feedback using a deployed network model.

FIGS. 26-30 depict graphs of experimental results using techniquesdisclosed herein.

FIG. 31A illustrates a flow diagram of an example method for imagereconstruction.

FIG. 31B provides further detail regarding a particular implementationof the example method of FIG. 31A for image reconstruction.

FIG. 32A illustrates an example implementation of the system designengine of FIG. 15A.

FIG. 32B illustrates an example implementation of the composite systemdigital model of FIG. 32A.

FIG. 33 illustrates a flow diagram of a method to monitor and improvesystem health, configuration, and/or design.

FIG. 34 illustrates an example representation of data flow between thesystem design engine deep learning network and a deep learning networkassociated with a device.

FIG. 35 is a block diagram of a processor platform structured to executethe example machine readable instructions to implement componentsdisclosed and described herein.

FIGS. 36-37 illustrate an example imaging system to which the methods,apparatus, and articles of manufacture disclosed herein can be applied.

The figures are not scale. Wherever possible, the same reference numberswill be used throughout the drawings and accompanying writtendescription to refer to the same or like parts.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of illustration specific examples that may be practiced. Theseexamples are described in sufficient detail to enable one skilled in theart to practice the subject matter, and it is to be understood thatother examples may be utilized and that logical, mechanical, electricaland other changes may be made without departing from the scope of thesubject matter of this disclosure. The following detailed descriptionis, therefore, provided to describe an exemplary implementation and notto be taken as limiting on the scope of the subject matter described inthis disclosure. Certain features from different aspects of thefollowing description may be combined to form yet new aspects of thesubject matter discussed below.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.

While certain examples are described below in the context of medical orhealthcare systems, other examples can be implemented outside themedical environment. For example, certain examples can be applied tonon-medical imaging such as non-destructive testing, explosivedetection, etc.

I. Overview

Imaging devices (e.g., gamma camera, positron emission tomography (PET)scanner, computed tomography (CT) scanner, X-Ray machine, magneticresonance (MR) imaging machine, ultrasound scanner, etc.) generatemedical images (e.g., native Digital Imaging and Communications inMedicine (DICOM) images) representative of the parts of the body (e.g.,organs, tissues, etc.) to diagnose and/or treat diseases. Medical imagesmay include volumetric data including voxels associated with the part ofthe body captured in the medical image. Medical image visualizationsoftware allows a clinician to segment, annotate, measure, and/or reportfunctional or anatomical characteristics on various locations of amedical image. In some examples, a clinician may utilize the medicalimage visualization software to identify regions of interest with themedical image.

Acquisition, processing, analysis, and storage of medical image dataplay an important role in diagnosis and treatment of patients in ahealthcare environment. A medical imaging workflow and devices involvedin the workflow can be configured, monitored, and updated throughoutoperation of the medical imaging workflow and devices. Machine learningcan be used to help configure, monitor, and update the medical imagingworkflow and devices.

Certain examples provide and/or facilitate improved imaging deviceswhich improve diagnostic accuracy and/or coverage. Certain examplesfacilitate improved image acquisition and reconstruction to provideimproved diagnostic accuracy. For example, image quality (IQ) metricsand automated validation can be facilitated using deep learning and/orother machine learning technologies.

Machine learning techniques, whether deep learning networks or otherexperiential/observational learning system, can be used to locate anobject in an image, understand speech and convert speech into text, andimprove the relevance of search engine results, for example. Deeplearning is a subset of machine learning that uses a set of algorithmsto model high-level abstractions in data using a deep graph withmultiple processing layers including linear and non-lineartransformations. While many machine learning systems are seeded withinitial features and/or network weights to be modified through learningand updating of the machine learning network, a deep learning networktrains itself to identify “good” features for analysis. Using amultilayered architecture, machines employing deep learning techniquescan process raw data better than machines using conventional machinelearning techniques. Examining data for groups of highly correlatedvalues or distinctive themes is facilitated using different layers ofevaluation or abstraction.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The term “deep learning” is a machine learningtechnique that utilizes multiple data processing layers to recognizevarious structures in data sets and classify the data sets with highaccuracy. A deep learning network can be a training network (e.g., atraining network model or device) that learns patterns based on aplurality of inputs and outputs. A deep learning network can be adeployed network (e.g., a deployed network model or device) that isgenerated from the training network and provides an output in responseto an input.

The term “supervised learning” is a deep learning training method inwhich the machine is provided already classified data from humansources. The term “unsupervised learning” is a deep learning trainingmethod in which the machine is not given already classified data butmakes the machine useful for abnormality detection. The term“semi-supervised learning” is a deep learning training method in whichthe machine is provided a small amount of classified data from humansources compared to a larger amount of unclassified data available tothe machine.

The term “representation learning” is a field of methods fortransforming raw data into a representation or feature that can beexploited in machine learning tasks. In supervised learning, featuresare learned via labeled input.

The term “convolutional neural networks” or “CNNs” are biologicallyinspired networks of interconnected data used in deep learning fordetection, segmentation, and recognition of pertinent objects andregions in datasets. CNNs evaluate raw data in the form of multiplearrays, breaking the data in a series of stages, examining the data forlearned features.

The term “transfer learning” is a process of a machine storing theinformation used in properly or improperly solving one problem to solveanother problem of the same or similar nature as the first. Transferlearning may also be known as “inductive learning”. Transfer learningcan make use of data from previous tasks, for example.

The term “active learning” is a process of machine learning in which themachine selects a set of examples for which to receive training data,rather than passively receiving examples chosen by an external entity.For example, as a machine learns, the machine can be allowed to selectexamples that the machine determines will be most helpful for learning,rather than relying only an external human expert or external system toidentify and provide examples.

The term “computer aided detection” or “computer aided diagnosis” referto computers that analyze medical images for the purpose of suggesting apossible diagnosis.

Deep Learning

Deep learning is a class of machine learning techniques employingrepresentation learning methods that allows a machine to be given rawdata and determine the representations needed for data classification.Deep learning ascertains structure in data sets using backpropagationalgorithms which are used to alter internal parameters (e.g., nodeweights) of the deep learning machine. Deep learning machines canutilize a variety of multilayer architectures and algorithms. Whilemachine learning, for example, involves an identification of features tobe used in training the network, deep learning processes raw data toidentify features of interest without the external identification.

Deep learning in a neural network environment includes numerousinterconnected nodes referred to as neurons. Input neurons, activatedfrom an outside source, activate other neurons based on connections tothose other neurons which are governed by the machine parameters. Aneural network behaves in a certain manner based on its own parameters.Learning refines the machine parameters, and, by extension, theconnections between neurons in the network, such that the neural networkbehaves in a desired manner.

Deep learning that utilizes a convolutional neural network segments datausing convolutional filters to locate and identify learned, observablefeatures in the data. Each filter or layer of the CNN architecturetransforms the input data to increase the selectivity and invariance ofthe data. This abstraction of the data allows the machine to focus onthe features in the data it is attempting to classify and ignoreirrelevant background information.

Deep learning operates on the understanding that many datasets includehigh level features which include low level features. While examining animage, for example, rather than looking for an object, it is moreefficient to look for edges which form motifs which form parts, whichform the object being sought. These hierarchies of features can be foundin many different forms of data such as speech and text, etc.

Learned observable features include objects and quantifiableregularities learned by the machine during supervised learning. Amachine provided with a large set of well classified data is betterequipped to distinguish and extract the features pertinent to successfulclassification of new data.

A deep learning machine that utilizes transfer learning may properlyconnect data features to certain classifications affirmed by a humanexpert. Conversely, the same machine can, when informed of an incorrectclassification by a human expert, update the parameters forclassification. Settings and/or other configuration information, forexample, can be guided by learned use of settings and/or otherconfiguration information, and, as a system is used more (e.g.,repeatedly and/or by multiple users), a number of variations and/orother possibilities for settings and/or other configuration informationcan be reduced for a given situation.

An example deep learning neural network can be trained on a set ofexpert classified data, for example. This set of data builds the firstparameters for the neural network, and this would be the stage ofsupervised learning. During the stage of supervised learning, the neuralnetwork can be tested whether the desired behavior has been achieved.

Once a desired neural network behavior has been achieved (e.g., amachine has been trained to operate according to a specified threshold,etc.), the machine can be deployed for use (e.g., testing the machinewith “real” data, etc.). During operation, neural networkclassifications can be confirmed or denied (e.g., by an expert user,expert system, reference database, etc.) to continue to improve neuralnetwork behavior. The example neural network is then in a state oftransfer learning, as parameters for classification that determineneural network behavior are updated based on ongoing interactions. Incertain examples, the neural network can provide direct feedback toanother process. In certain examples, the neural network outputs datathat is buffered (e.g., via the cloud, etc.) and validated before it isprovided to another process.

Deep learning machines using convolutional neural networks (CNNs) can beused for image analysis. Stages of CNN analysis can be used for facialrecognition in natural images, computer-aided diagnosis (CAD), etc.

High quality medical image data can be acquired using one or moreimaging modalities, such as x-ray, computed tomography (CT), molecularimaging and computed tomography (MICT), magnetic resonance imaging(MRI), etc. Medical image quality is often not affected by the machinesproducing the image but the patient. A patient moving during an MRI cancreate a blurry or distorted image that can prevent accurate diagnosis,for example.

Interpretation of medical images, regardless of quality, is only arecent development. Medical images are largely interpreted byphysicians, but these interpretations can be subjective, affected by thecondition of the physician's experience in the field and/or fatigue.Image analysis via machine learning can support a healthcarepractitioner's workflow.

Deep learning machines can provide computer aided detection support toimprove their image analysis with respect to image quality andclassification, for example. However, issues facing deep learningmachines applied to the medical field often lead to numerous falseclassifications. Deep learning machines must overcome small trainingdatasets and require repetitive adjustments, for example.

Deep learning machines, with minimal training, can be used to determinethe quality of a medical image, for example. Semi-supervised andunsupervised deep learning machines can be used to quantitativelymeasure qualitative aspects of images. For example, deep learningmachines can be utilized after an image has been acquired to determineif the quality of the image is sufficient for diagnosis. Supervised deeplearning machines can also be used for computer aided diagnosis.Supervised learning can help reduce susceptibility to falseclassification, for example.

Deep learning machines can utilize transfer learning when interactingwith physicians to counteract the small dataset available in thesupervised training. These deep learning machines can improve theircomputer aided diagnosis over time through training and transferlearning.

II. Description of Examples

Example Deep Learning Network Systems

FIG. 1 is a representation of an example deep learning neural network100. The example neural network 100 includes layers 120, 140, 160, and180. The layers 120 and 140 are connected with neural connections 130.The layers 140 and 160 are connected with neural connections 150. Thelayers 160 and 180 are connected with neural connections 170. Data flowsforward via inputs 112, 114, 116 from the input layer 120 to the outputlayer 180 and to an output 190.

The layer 120 is an input layer that, in the example of FIG. 1, includesa plurality of nodes 122, 124, 126. The layers 140 and 160 are hiddenlayers and include, the example of FIG. 1, nodes 142, 144, 146, 148,162, 164, 166, 168. The neural network 100 may include more or lesshidden layers 140 and 160 than shown. The layer 180 is an output layerand includes, in the example of FIG. 1A, a node 182 with an output 190.Each input 112-116 corresponds to a node 122-126 of the input layer 120,and each node 122-126 of the input layer 120 has a connection 130 toeach node 142-148 of the hidden layer 140. Each node 142-148 of thehidden layer 140 has a connection 150 to each node 162-168 of the hiddenlayer 160. Each node 162-168 of the hidden layer 160 has a connection170 to the output layer 180. The output layer 180 has an output 190 toprovide an output from the example neural network 100.

Of connections 130, 150, and 170 certain example connections 132, 152,172 may be given added weight while other example connections 134, 154,174 may be given less weight in the neural network 100. Input nodes122-126 are activated through receipt of input data via inputs 112-116,for example. Nodes 142-148 and 162-168 of hidden layers 140 and 160 areactivated through the forward flow of data through the network 100 viathe connections 130 and 150, respectively. Node 182 of the output layer180 is activated after data processed in hidden layers 140 and 160 issent via connections 170. When the output node 182 of the output layer180 is activated, the node 182 outputs an appropriate value based onprocessing accomplished in hidden layers 140 and 160 of the neuralnetwork 100.

FIG. 2 illustrates a particular implementation of the example neuralnetwork 100 as a convolutional neural network 200. As shown in theexample of FIG. 2, an input 110 is provided to the first layer 120 whichprocesses and propagates the input 110 to the second layer 140. Theinput 110 is further processed in the second layer 140 and propagated tothe third layer 160. The third layer 160 categorizes data to be providedto the output layer 180. More specifically, as shown in the example ofFIG. 2, a convolution 204 (e.g., a 5×5 convolution, etc.) is applied toa portion or window (also referred to as a “receptive field”) 202 of theinput 110 (e.g., a 32×32 data input, etc.) in the first layer 120 toprovide a feature map 206 (e.g., a (6×) 28×28 feature map, etc.). Theconvolution 204 maps the elements from the input 110 to the feature map206. The first layer 120 also provides subsampling (e.g., 2×2subsampling, etc.) to generate a reduced feature map 210 (e.g., a (6×)14×14 feature map, etc.). The feature map 210 undergoes a convolution212 and is propagated from the first layer 120 to the second layer 140,where the feature map 210 becomes an expanded feature map 214 (e.g., a(16×) 10×10 feature map, etc.). After subsampling 216 in the secondlayer 140, the feature map 214 becomes a reduced feature map 218 (e.g.,a (16×) 4×5 feature map, etc.). The feature map 218 undergoes aconvolution 220 and is propagated to the third layer 160, where thefeature map 218 becomes a classification layer 222 forming an outputlayer of N categories 224 with connection 226 to the convoluted layer222, for example.

FIG. 3 is a representation of an example implementation of an imageanalysis convolutional neural network 300. The convolutional neuralnetwork 300 receives an input image 302 and abstracts the image in aconvolution layer 304 to identify learned features 310-322. In a secondconvolution layer 330, the image is transformed into a plurality ofimages 330-338 in which the learned features 310-322 are eachaccentuated in a respective sub-image 330-338. The images 330-338 arefurther processed to focus on the features of interest 310-322 in images340-348. The resulting images 340-348 are then processed through apooling layer which reduces the size of the images 340-348 to isolateportions 350-354 of the images 340-348 including the features ofinterest 310-322. Outputs 350-354 of the convolutional neural network300 receive values from the last non-output layer and classify the imagebased on the data received from the last non-output layer. In certainexamples, the convolutional neural network 300 may contain manydifferent variations of convolution layers, pooling layers, learnedfeatures, and outputs, etc.

FIG. 4A illustrates an example configuration 400 to apply a deeplearning network to process and/or otherwise evaluate an image. Deeplearning can be applied to a variety of processes including imageacquisition, image reconstruction, image analysis/diagnosis, etc. Asshown in the example configuration 400 of FIG. 4A, raw data 410 (e.g.,raw data 410 such as sonogram raw data, etc., obtained from an imagingscanner such as an x-ray, computed tomography, ultrasound, magneticresonance, etc., scanner) is fed into a deep learning network 420. Thedeep learning network 420 processes the data 410 to correlate and/orotherwise combine the raw image data 420 into a resulting image 430(e.g., a “good quality” image and/or other image providing sufficientquality for diagnosis, etc.). The deep learning network 420 includesnodes and connections (e.g., pathways) to associate raw data 410 with afinished image 430. The deep learning network 420 can be a training deeplearning network that learns the connections and processes feedback toestablish connections and identify patterns, for example. The deeplearning network 420 can be a deployed deep learning network that isgenerated from a training network and leverages the connections andpatterns established in the training network to take the input raw data410 and generate the resulting image 430, for example.

Once the DLN 420 is trained and produces good images 630 from the rawimage data 410, the network 420 can continue the “self-learning” processand refine its performance as it operates. For example, there is“redundancy” in the input data (raw data) 410 and redundancy in thenetwork 420, and the redundancy can be exploited.

If weights assigned to nodes in the DLN 420 are examined, there arelikely many connections and nodes with very low weights. The low weightsindicate that these connections and nodes contribute little to theoverall performance of the DLN 420. Thus, these connections and nodesare redundant. Such redundancy can be evaluated to reduce redundancy inthe inputs (raw data) 410. Reducing input 410 redundancy can result insavings in scanner hardware, reduced demands on components, and alsoreduced exposure dose to the patient, for example.

In deployment, the configuration 400 forms a package 400 including aninput definition 410, a trained network 420, and an output definition430. The package 400 can be deployed and installed with respect toanother system, such as an imaging system, analysis engine, etc.

As shown in the example of FIG. 4B, the deep learning network 420 can bechained and/or otherwise combined with a plurality of deep learningnetworks 421-423 to form a larger learning network. The combination ofnetworks 420-423 can be used to further refine responses to inputsand/or allocate networks 420-423 to various aspects of a system, forexample.

In some examples, in operation, “weak” connections and nodes caninitially be set to zero. The DLN 420 then processes its nodes in aretaining process. In certain examples, the nodes and connections thatwere set to zero are not allowed to change during the retraining. Giventhe redundancy present in the network 420, it is highly likely thatequally good images will be generated. As illustrated in FIG. 4B, afterretraining, the DLN 420 becomes DLN 421. DLN 421 is also examined toidentify weak connections and nodes and set them to zero. This furtherretrained network is DLN 422. The example DLN 422 includes the “zeros”in DLN 421 and the new set of nodes and connections. The DLN 422continues to repeat the processing until a good image quality is reachedat a DLN 423, which is referred to as a “minimum viable net (MVN)”. TheDLN 423 is a MVN because if additional connections or nodes areattempted to be set to zero in DLN 423, image quality can suffer.

Once the MVN has been obtained with the DLN 423, “zero” regions (e.g.,dark irregular regions in a graph) are mapped to the input 410. Eachdark zone is likely to map to one or a set of parameters in the inputspace. For example, one of the zero regions may be linked to the numberof views and number of channels in the raw data. Since redundancy in thenetwork 423 corresponding to these parameters can be reduced, there is ahighly likelihood that the input data can be reduced and generateequally good output. To reduce input data, new sets of raw data thatcorrespond to the reduced parameters are obtained and run through theDLN 421. The network 420-423 may or may not be simplified, but one ormore of the DLNs 420-423 is processed until a “minimum viable input(MVI)” of raw data input 410 is reached. At the MVI, a further reductionin the input raw data 410 may result in reduced image 430 quality. TheMVI can result in reduced complexity in data acquisition, less demand onsystem components, reduced stress on patients (e.g., less breath-hold orcontrast), and/or reduced dose to patients, for example.

By forcing some of the connections and nodes in the DLNs 420-423 tozero, the network 420-423 to build “collaterals” to compensate. In theprocess, insight into the topology of the DLN 420-423 is obtained. Notethat DLN 421 and DLN 422, for example, have different topology sincesome nodes and/or connections have been forced to zero. This process ofeffectively removing connections and nodes from the network extendsbeyond “deep learning” and can be referred to as “deep-deep learning”.

In certain examples, input data processing and deep learning stages canbe implemented as separate systems. However, as separate systems,neither module may be aware of a larger input feature evaluation loop toselect input parameters of interest/importance. Since input dataprocessing selection matters to produce high-quality outputs, feedbackfrom deep learning systems can be used to perform input parameterselection optimization or improvement via a model. Rather than scanningover an entire set of input parameters to create raw data (e.g., whichis brute force and can be expensive), a variation of active learning canbe implemented. Using this variation of active learning, a startingparameter space can be determined to produce desired or “best” resultsin a model. Parameter values can then be randomly decreased to generateraw inputs that decrease the quality of results while still maintainingan acceptable range or threshold of quality and reducing runtime byprocessing inputs that have little effect on the model's quality.

FIG. 5 illustrates example training and deployment phases of a deeplearning network. As shown in the example of FIG. 5, in the trainingphase, a set of inputs 502 is provided to a network 504 for processing.In this example, the set of inputs 502 can include facial features of animage to be identified. The network 504 processes the input 502 in aforward direction 506 to associate data elements and identify patterns.The network 504 determines that the input 502 represents a dog 508. Intraining, the network result 508 is compared 510 to a known outcome 512.In this example, the known outcome 512 is a human face (e.g., the inputdata set 502 represents a human face, not a dog face). Since thedetermination 508 of the network 504 does not match 510 the knownoutcome 512, an error 514 is generated. The error 514 triggers ananalysis of the known outcome 512 and associated data 502 in reversealong a backward pass 516 through the network 504. Thus, the trainingnetwork 504 learns from forward 506 and backward 516 passes with data502, 512 through the network 405.

Once the comparison of network output 508 to known output 512 matches510 according to a certain criterion or threshold (e.g., matches ntimes, matches greater than x percent, etc.), the training network 504can be used to generate a network for deployment with an externalsystem. Once deployed, a single input 520 is provided to a deployed deeplearning network 522 to generate an output 524. In this case, based onthe training network 504, the deployed network 522 determines that theinput 520 is an image of a human face 524.

FIG. 6 illustrates an example product leveraging a trained networkpackage to provide a deep learning product offering. As shown in theexample of FIG. 6, an input 610 (e.g., raw data) is provided forpreprocessing 620. For example, the raw input data 610 is preprocessed620 to check format, completeness, etc. Once the data 610 has beenpreprocessed 620, patches are created 630 of the data. For example,patches or portions or “chunks” of data are created 630 with a certainsize and format for processing. The patches are then fed into a trainednetwork 640 for processing. Based on learned patterns, nodes, andconnections, the trained network 640 determines outputs based on theinput patches. The outputs are assembled 650 (e.g., combined and/orotherwise grouped together to generate a usable output, etc.). Theoutput is then displayed 660 and/or otherwise output to a user (e.g., ahuman user, a clinical system, an imaging modality, a data storage(e.g., cloud storage, local storage, edge device, etc.), etc.).

As discussed above, deep learning networks can be packaged as devicesfor training, deployment, and application to a variety of systems. FIGS.7A-7C illustrate various deep learning device configurations. Forexample, FIG. 7A shows a general deep learning device 700. The exampledevice 700 includes an input definition 710, a deep learning networkmodel 720, and an output definitions 730. The input definition 710 caninclude one or more inputs translating into one or more outputs 730 viathe network 720.

FIG. 7B shows an example training deep learning network device 701. Thatis, the training device 701 is an example of the device 700 configuredas a training deep learning network device. In the example of FIG. 7B, aplurality of training inputs 711 are provided to a network 721 todevelop connections in the network 721 and provide an output to beevaluated by an output evaluator 731. Feedback is then provided by theoutput evaluator 731 into the network 721 to further develop (e.g.,train) the network 721. Additional input 711 can be provided to thenetwork 721 until the output evaluator 731 determines that the network721 is trained (e.g., the output has satisfied a known correlation ofinput to output according to a certain threshold, margin of error,etc.).

FIG. 7C depicts an example deployed deep learning network device 703.Once the training device 701 has learned to a requisite level, thetraining device 701 can be deployed for use. While the training device701 processes multiple inputs to learn, the deployed device 703processes a single input to determine an output, for example. As shownin the example of FIG. 7C, the deployed device 703 includes an inputdefinition 713, a trained network 723, and an output definition 733. Thetrained network 723 can be generated from the network 721 once thenetwork 721 has been sufficiently trained, for example. The deployeddevice 703 receives a system input 713 and processes the input 713 viathe network 723 to generate an output 733, which can then be used by asystem with which the deployed device 703 has been associated, forexample.

In certain examples, the training device 701 and/or deployed device 703can be integrated in a learning and improvement factory to provideoutput to a target system, collect feedback, and update/re-train basedon the feedback. FIG. 8A illustrates an example learning and improvementfactory 800 including the training deep learning device 701 and thedeployed deep learning device 703. As shown in the example of FIG. 8A,the training deep learning device 701 provides output to a modelevaluator 802. The model evaluator 802 compares the output of the device701 to a known output and/or otherwise measures accuracy, precision,and/or quality of the output to determine whether or not the trainingdevice 701 is ready for deployment. Once the model evaluator 802 hasdetermined that the device 701 has been properly trained, the modelevaluator 802 provides a model of the trained network from the device701 to a model deployment module 804, which prepares the trained modelfor deployment. The module 804 provides the prepared model to a deployeddeep learning device generator 806 which instantiates the deployed deeplearning device 703 with a framework or package for input definition andoutput definition around a model of the trained network from the device701.

The deployed device 703 operates on input and provides output, and afeedback collector 808 monitors the output (and input) and gathersfeedback based on operation of the deployed deep learning device 703.The feedback is stored in feedback storage 810 until a certain amount offeedback has been collected (e.g., a certain quantity, a certainquality/consistency, a certain time period, etc.). Once sufficientfeedback has been collected, a re-training initiator 812 is triggered.The re-training initiator 812 retrieves data from the feedback storage810 and operates in conjunction with a re-training data selector 814 toselect data from the feedback storage 810 to provide to the trainingdeep learning device 701. The network of the training device 701 is thenupdated/re-trained using the feedback until the model evaluator 802 issatisfied that the training network model is complete. Theupdated/re-trained model is then prepared and deployed in the deployeddeep learning device 703 as described above.

As shown in the example of FIG. 8B, the learning and improvement factory800 can be implemented in a variety of levels/hierarchy. For example, amicro learning and improvement factory 801 may model and/or providesupport for a particular device, device feature, etc. A learning andimprovement factory 803 can target an overall system or installation,for example. A global learning and improvement factory 805 can provideoutput and model an organization, facility, etc. Thus, learning andimprovement factories 801-805 can be implemented through an organizationto learn, model, and improve system accuracy, performance,effectiveness, safety, efficiency, etc.

FIG. 8C illustrates a flow diagram of an example method 820 to train anddeploy a deep learning network model. At block 822, a deep learningnetwork model is trained. For example, a plurality of inputs areprovided to the network, and output is generated. At block 824, the deeplearning network model is evaluated. For example, output of the networkis compared against known/reference output for those inputs. As thenetwork makes connections and learns, the accuracy of the network modelimproves. At block 826, the output is evaluated to determine whether thenetwork has successfully modeled the expected output. If the network hasnot, then the training process continues at block 822. If the networkhas successfully modeled the output, then, at block 828, a deep learningmodel-based device is generated. At block 830, the deep learning deviceis deployed.

At block 832, feedback from operation of the deployed deep learningmodel-based device is collected and stored until the collected feedbacksatisfies a threshold (block 834). Feedback can include input, deployedmodel information, pre- and/or post-processing information, actualand/or corrected output, etc. Once the feedback collection threshold issatisfied, at block 836, model re-training is initiated. At block 838,data from the collected feedback (and/or other input data) is selectedto re-train the deep learning model. Data selection can include pre-and/or post-processing to properly format the data for model training,etc. Control then passes to block 822 to (re)train the deep learningnetwork model.

FIG. 8D re-iterates an example process to collect and store feedbackduring operation 840 of the deployed deep learning network model-baseddevice and re-train the model for re-deployment. Feedback can includeinput, deployed model information, pre- and/or post-processinginformation, actual and/or corrected output, etc. At block 842, thecollected feedback is reviewed to determine whether the collectedfeedback satisfies a collection/feedback threshold (e.g., an amount offeedback, a frequency of feedback, a type of feedback, an amount of timelapsed for feedback, etc.). If the threshold is not satisfied, thenfeedback collection and storage continues at block 840. If the thresholdis satisfied, however, then, at block 844, model re-training isinitiated.

At block 846, data is selected to re-train the deep learning networkmodel. Data includes collected feedback and can also include other dataincluding original input data to the model and/or other reference data,for example. Thus, the model may not be re-trained exclusively onfeedback data but on a mix of old and new data fed into the deeplearning model, for example. Data selection can include pre- and/orpost-processing to properly format the data for model training, etc.

At block 848, the deep learning network model is (re) trained. That is,data is provided as input to modify the network model and generate anoutput. At block 850, the output is evaluated to determine whether thenetwork model has been (re)trained. At block 852, if the network has notmodeled expected output, then control reverts to block 848 to continuemodel training with input and output evaluation. If the (re)trainednetwork has successfully modeled the expected output (e.g., over acertain threshold of times, etc.), then, at block 854, the deep learningmodel-based device is generated. At block 856, the deep learning deviceis deployed. Thus, a model can be initiated trained and/or re-trainedand used to generated a deployed network model-based device. While thedeployed device is not modified during operation, the training model canbe updated and/or otherwise modified and periodically used toreplace/re-deploy the deployed network model, for example.

FIG. 9 illustrates an example system including a data factory 902,application factory 916, and learning factory 924 leveraging deeplearning to provide applications for one or more systems and/orassociated users. In the example of FIG. 9, the data factory 902includes one or more data schema 904, curation tools 906, bulk dataingestion 908, data selection/filter 910, continuous data ingestion 912,and data catalog/lake 914. The example data factory 902 ingests data908, 912 and can process the data to select/filter the data 910 andformat the data according to a certain schema 904. The data can beorganized according to one or more curation tools 906 and stored in thedata catalog/lake 914 to be made available to the application factory916 and/or the learning factory 924. The application factory 916includes a viewer 918 allowing a system and/or associated user to viewand/or access applications available via application services 920 and/ora pipelines catalog 922.

In the example of FIG. 9, the learning factory 924 includes a modelcatalog 926 including one or more network models (e.g., deeplearning-based network models, machine learning-based network machines,etc.) available to the application factory 916 and/or other externalsystem, for example. The learning factory 924 also includes data science928 including data to form and/or be leveraged by models in the modelcatalog 926. The example data science 928 includes an architecturecatalog 930, data preparation 932, results/reporting 934, training andvalidation 936, and testing 938 to organize and otherwise pre-processdata, train and validate a learning network, report results, and testoutcomes, etc. Trained and validated networks are made available fordeployment in one or more applications via the model catalog 926, forexample.

FIG. 10 illustrates an overview of a medical device ecosystem 1000including devices physically (physical factory) deployed internally 1002and externally 1004 with a digital factory 1006. As shown in the exampleof FIG. 10, the digital factory 1006 includes a data factory 902, datacatalog 914, learning factory 924, deep learning network-based modelcatalog 926, etc. The digital factory 1006 provides and/or interactswith one or more digital models 1008 (e.g., deep learning networkmodels, machine learning models, etc.). The digital factory 1006interacts with a physical factory including a plurality of devices1010-1016 deployed internally 1002 (e.g., devices 1010 and 1012 andexternally 1004 (e.g., devices 10140 and 1016). Devices 1010-1016 areconnected to the digital factory 1006 and can upload data to the digitalfactory 1006, subscribe to model(s) from the catalog 926, update models,etc. Devices 1010, 1012 in internal deployment 1002 can be used fortesting and refinement purposes with the digital factory 1006, forexample, while devices 1014, 1016 in external deployment 1004 are “live”with deployed models aiding devices 1014, 1016 in decision-making and/orother execution, for example.

FIG. 11 illustrates an example configuration 1100 of the physical device1010 and its data flow interacting with the digital factory 1006, whichcan include the data factory 902 and its data catalog 914, data curation906, as well as the learning factory 924 and its model catalog 926, andthe application factory 916 with its application pipelines catalog 922,etc. As shown in the example of FIG. 11, the physical device 1010 (e.g.,an imaging scanner) includes a device controller 1102, a detector 1104,and a source 1106 to acquire image data of a patient 1108. The scannerdevice 1010 provides a scan context 1110 and scanner data 1112 in imageacquisition 1114. The acquisition engine 1114 interacts with the digitalfactory 1006 to model acquisition of image data, etc. Acquired imagedata is provided for reconstruction 1116, and the reconstruction engine1116 also interacts with the digital factory 1006 for model-basedresources for reconstruction of the acquired image data. Thereconstructed image is provided for viewing 1118 in conjunction with anapplication provided from the digital factory 1006, for example. One ormore applications and/or measurements 1120 can be applied to thereconstructed image (e.g., based on models and/or other applicationsfrom the digital factory 1006, etc.), for example. Processed imageand/or other data can be leveraged in one or more clinical workflows1122, which in turn leverage applications, data, and models from thedigital factory 1006 to facilitate improved and/or automated executionof the clinical workflow(s) 1122. Outcome(s) of the workflow(s) can beprovided to analytics and decision support 1124 to drive conclusion(s),recommendation(s), next action(s), model refinement, etc., inconjunction with the digital factory 1006, for example.

FIG. 12 illustrates a flow diagram of an example method 1200 to processand leverage data in the data factory 902 and learning factory 924. Atblock 1202, data is ingested (e.g., by bulk 908 and/or continuous 912ingestion, etc.). At block 1204, the ingested data is curated. Forexample, one or more data categorization, processing, and/or othercuration tools 906 can be applied to organize the ingested data. Atblock 1206, the curated data is processed and used for learning. Forexample, the curated data can be analyzed, used to train a deep learningnetwork, etc. At block 1208, output generated from the processed dataand based on the learning is packaged and deployed. For example, one ormore trained deep learning networks can be cataloged in the modelcatalog 926 and made available for deployment.

FIG. 13 provides further detail regarding the example method 1200 toprocess and leverage data in the data factory 902 and learning factory924. As shown in the example of FIG. 13, data ingestion 1202 includesextracting data from one or more on-premise data sources 1302 such as apicture archiving and communication system (PACS), vendor-neutralarchive (VNA), enterprise archive (EA), imaging scanner, etc. Ingesteddata is collected and stored 1304 in a data catalog/lake 1306. At block1308, data is selected and/or fetched for viewing 1310 via the imageand/or other data viewer 1310.

FIG. 13 also provides further detail regarding datacuration/organization 1204. At block 1312, the selected/fetched data isanalyzed to determine if the correct data for the application and/orother request. If not, control reverts to block 1308 to select/fetchdifferent data. If the right data has been selected, then, at block1314, the data is reviewed to determine whether or not the data iscurated. If the data is not curated, then, at block 1316, data curationoccurs. For example, data curation involves accurate labeling of data,identification of a region of interest (ROI) with editable bounding box,addition of meta data information, modifying improper pre-curationinformation, etc., and saving as a new data set. Curated data isprovided back to the data catalog 1306. If, at block 1314, the data iscurated, then, control shifts to data processing at block 1206.

As shown in more detail in FIG. 13, data processing 1206 includespreparing the data 1318 using one or more data preparation tools 1320.The data is prepared for development of artificial intelligence (AI)(block 1322), such as development of a deep learning network modeland/or other machine learning model, etc. Data preparation 1318 (e.g.,for training, validation, testing, etc.) includes creation and labelingof data patches, image processing (e.g., crop, squash, etc.), dataaugmentation to generate more training samples, three-dimensional imageprocessing to provide to a learning network model, database creation andstorage (e.g., json and/or other format), patch image data storage(e.g., .png, .jpeg, etc.), etc. In some examples, a final patch imagedata dataset is stored in the data catalog/lake 1306.

At block 1324, an AI methodology (e.g., deep learning network modeland/or other machine learning model, etc.) is selected from an AIcatalog 1326 of available models, for example. For example, a deeplearning model can be imported, the model can be modified, transferlearning can be facilitated, an activation function can be selectedand/or modified, machine learning selection and/or improvement can occur(e.g., support vector machine (SVM), random forest (RF), etc.), anoptimization algorithm (e.g., stochastic gradient descent (SGD), AdaG,etc.) can be selected and/or modified, etc. The AI catalog 1326 caninclude one or more AI models such as good old fashioned artificialintelligence (GOFAI) (e.g., expert systems, etc.), machine learning (ML)(e.g., SVM, RF, etc.), deep learning (DL) (e.g., convolutional neuralnetwork (CNN), recurrent neural network (RNN), long short-term memory(LS™), generative adversarial network (GAN), etc.), paradigms (e.g.,supervised, unsupervised, reinforcement, etc.), etc.

At block 1328, model development is initialized (e.g., using anactivation function, weight, bias, hyper-parameters, etc.), and, atblock 1330, training of the model occurs (e.g., as described above,etc.). In certain examples, training 1330 is an iterative processincluding training and validation involving hyper-parameter setup,hyper-parameter search, training/validation set accuracy graph(s), areaunder the curve (AUC) graphing, intermittent model generating andsaving, early and/or manually stop training, etc. At block 1332, anaccuracy of the AI model is evaluated to determine whether the accuracyis acceptable. If the accuracy is not acceptable, then control revertsto block 1318 for additional data preparation and subsequentdevelopment. If the accuracy is acceptable, then, at block 1334, the AImodel is released for testing (e.g., providing additional input(s) andevaluating output(s), etc.). At block 1336, results of the testing arereported. For example, a continuous recording of experimental parametersand outcomes can be provided.

FIG. 13 also provides further example detail regarding packaging anddeployment 1208. At block 1338, if the accuracy of the tested model isnot acceptable, control reverts to block 1318 for data preparation. Ifthe accuracy of the tested model is acceptable, then, at block 1340, themodel is added to a catalog of trained models. At block 1342, one ormore of the models in the catalog of trained models is packaged, and, atblock 1344, the package is deployed (e.g., to a target site, targetsystem, etc.).

Example Improved Healthcare Systems Utilizing Deep and/or Other MachineLearning and Associated Methods

FIG. 14 illustrates an example healthcare system 1400 for patientevaluation and diagnosis. The example system 1400 includes an imagingdevice 1410, an information subsystem 1420, an acquisition engine 1430,a reconstruction engine 1440, and a diagnosis engine 1450 forinteraction with humans 1402 such as a user 1404 (e.g., a physician,nurse, technician, and/or other healthcare practitioner, etc.) and apatient 1406. The components of the healthcare system 1400 can beimplemented using one or more processors executing hardcodedconfiguration, firmware configuration, software instructions inconjunction with a memory, etc. For example, one or more components ofthe system 1400 can include a processor-based system including acombination of hardware and/or software code, routines, modules, orinstructions adapted to perform the presently discussed functionality,including performance of various elements of the methods describedelsewhere herein. It should be noted that such software routines may beembodied in a manufacture (e.g., a compact disc, a hard drive, a flashmemory, a universal serial bus (USB)-based drive, random access memory(RAM), read only memory (ROM), etc.) and configured to be executed by aprocessor to effect performance of the functionality described herein.

Using the example system 1400, the patient 1404 can be examined by theimaging system 1410 (e.g., CT, x-ray, MR, PET, ultrasound, MICT, singlephoton emission computed tomography (SPECT), digital tomosynthesis,etc.) based on settings from the information subsystem 1420 and/oracquisition engine 1430. Settings can be dictated and/or influenced by adeployed deep learning network model/device, such as CNN, RNN, etc.Based on information, such as a reason for exam, patient identification,patient context, population health information, etc., imaging device1410 settings can be configured for image acquisition with respect tothe patient 1406 by the acquisition engine 1430, alone or in conjunctionwith the information subsystem 1420 (e.g., a picture archiving andcommunication system (PACS), hospital information system (HIS),radiology information system (RIS), laboratory information system (LIS),cardiovascular information system (CVIS), etc.). The information fromthe information subsystem 1420 and/or acquisition engine 1430, as wellas feedback from the imaging device 1410, can be collected and providedto a training deep learning network model to modify future settings,recommendations, etc., for image acquisition, for example. Periodicallyand/or upon satisfaction of certain criterion, the training deeplearning network model can process the feedback and generate an updatedmodel for deployment with respect to the system 1400.

Acquired or raw image data from the imaging device 1410, alone or inconjunction with additional patient history, patient context, populationhealth information, reason for exam, etc., is provided to thereconstruction engine 1440 to process the data to generate a resultingimage. The reconstruction engine 1440 uses the information and acquiredimage data to reconstruct one or more two-dimensional (2D) and/orthree-dimensional (3D) images of the patient 1406. Method(s) forreconstruction, reconstruction engine 1440 setting(s), etc., can be setand/or influenced by a deep learning network, such as a CNN, RNN, etc.For example, slice thickness, image quality, etc., can be determined andmodified using a deep learning network.

In certain examples, raw image data can be preprocessed by thereconstruction engine 1440. Preprocessing may include one or moresub-processes, such as intensity correction, resembling, filtering, etc.In certain examples, anatomical markers in the image data may bedetected, and an image grid may be created. Based on the anatomicalmarkers and the image grid, the reconstruction engine can register theimage data (e.g., according to a reference coordinate system, etc.).Following registration, features of interest in the image data may beextracted.

In certain examples, particular features that are of interest in the rawimage data may vary depending on a particular disease or condition ofinterest. For example, in diagnosing neurological conditions, it may beuseful to extract certain features of brain image data to facilitatediagnosis. Further, in some examples, it may be desirable to determinethe thickness of the cerebral cortex of a patient or of one or morereference individuals.

Certain examples process raw data acquired by the imaging device 1410and acquisition engine 1430 and provide the raw image data to thereconstruction engine 1440 to produce one or both of a) amachine-readable image provided to the diagnostic decision supportengine 1450 and b) a human-viewable image displayed for user diagnosis.

For example, while image reconstruction is primarily performed for humanconsumption, pre-reconstruction data can be used by a machine which doesnot care whether or not data has been reconstructed to be viewable by ahuman. Thus, pre-reconstruction data can be processed differently forhuman consumption and for machine consumption. Machine-readable imagedata can be processed by the reconstruction engine 1440 according toindicators of a given disease, for example, so that the reconstructionengine 1440 and/or the diagnosis engine 1450 can identify patternsindicative of the disease without performing reconstruction (e.g., inthe raw image data acquisition state). Thus, in some examples, thereconstruction engine 1440 can perform a diagnosis with the diagnosisengine 1450 rather than relying on the user 1404 to interact with thediagnosis engine 1450 to make a clinical diagnosis.

Image output from the reconstruction engine 1440 can then be provided tothe diagnosis engine 1450. The diagnosis engine 1450 can take image datafrom the reconstruction engine 1440 and/or non-image data from theinformation subsystem 1420 and process the data (e.g., static data,dynamic data, longitudinal data, etc.) to determine a diagnosis (and/orfacilitate a diagnosis by the user 1404) with respect to the patient1406. Data provided to the diagnosis engine 1450 can also include datafrom one or more patient monitors, such as an electroencephalography(EEG) device, an electrocardiography (ECG or EKG) device, anelectromyography (EMG) device, an electrical impedance tomography (EIT)device, an electronystagmography (ENG) device, a device adapted tocollect nerve conduction data, and/or some combination of these devices.

In some examples, the diagnosis engine 1450 processes one or morefeatures of interest from the image data to facilitate diagnosis of thepatient 1406 with respect to one or more disease types or diseaseseverity levels. Image data may be obtained from various sources, suchas the imaging device 1410, the information subsystem 1420, otherdevice, other database, etc. Further, such image data may be related toa particular patient, such as the patient 1406, or to one or morereference individuals of a population sample. The image data can beprocessed by the reconstruction engine 1440 and/or the diagnosis engine1450 to register and extract features of interest from the image, forexample.

Information can then be output from the diagnosis engine 1450 to theuser 1404, the information subsystem 1420, and/or other system forfurther storage, transmission, analysis, processing, etc. Informationcan be displayed in alphanumeric data format and tabulated for furtheranalysis and review (e.g., based on metric analysis, deviation metric,historical reference comparison, etc.), for example. Alternatively or inaddition, data can be presented holistically for analysis via heat map,deviation map, surface matrix, etc., taken alone or with respect toreference data, for example. U.S. Pat. Nos. 9,271,651, 8,934,685,8,430,816, 8,099,299, and 8,010,381, commonly owned by the presentassignee, provide further disclosure regarding an example holisticanalysis.

A patient diagnosis can be provided with respect to various patientdisease types and/or patient conditions, as well as associated severitylevels, while also providing decision support tools for user-diagnosisof patients. For example, patient clinical image and non-imageinformation can be visualized together in a holistic, intuitive, anduniform manner, facilitating efficient diagnosis by the user 1404. Inanother example, patient cortical deviation maps and reference corticaldeviation maps of known brain disorders can be visualized along withcalculation of additional patient and reference deviation maps, and thecombination of such maps with other clinical tests, to enablequantitative assessment and diagnosis of brain disorders.

Making a diagnosis is a very specialized task, and even highly-trainedmedical image experts conduct a subjective evaluation of an image. Dueto this inherent subjectivity, diagnoses can be inconsistent andnon-standardized. The diagnosis engine 1450 can employ a deep learningnetwork, such as a CNN, RNN, etc., to help improve consistency,standardization, and accuracy of diagnoses. Additional data, such asnon-image data, can be included in the deep learning network by thediagnosis engine 1450 to provide a holistic approach to patientdiagnosis.

In certain examples, the components of the system 1400 can communicateand exchange information via any type of public or private network suchas, but not limited to, the Internet, a telephone network, a local areanetwork (LAN), a cable network, and/or a wireless network. To enablecommunication via the network, one or more components of the system 1400includes a communication interface that enables a connection to anEthernet, a digital subscriber line (DSL), a telephone line, a coaxialcable, or any wireless connection, etc.

In certain examples, the information subsystem 1420 includes a localarchive and a remote system. The remote system periodically and/or upona trigger receives the local archive via the network. The remote systemmay gather local archives (e.g., including the local archive from theinformation subsystem 1420, reconstruction engine 1440, diagnosis engine1450, etc.) from various computing devices to generate a database ofremote medical image archives. In some examples, the remote systemincludes a machine learning algorithm to analyze, correlate, and/orprocess information to develop large data analytics based on archivesfrom various clinical sites based. For example, a plurality of imagescan be gathered by the remote system to train and test a neural networkto be deployed to automatically detect regions of interest in images(e.g., auto-contour, etc.).

FIG. 15 illustrates a further detailed view of an example improvedhealthcare system 1500 for patient evaluation and diagnosis. In theexample of FIG. 15, the imaging device 1410, information system 1420,acquisition engine 1430, reconstruction engine 1440, and diagnosisengine 1450 are configured with a plurality of deep learning networks1522, 1532, 1542, a system health module 1550, and a system designengine 1560.

As shown in the example of FIG. 15A, each of the acquisition engine1430, reconstruction engine 1440, and diagnosis engine 1450 communicateswith an associated learning and improvement factory 1520, 1530, 1540 forfeedback evaluation and training and, also, includes a deployed deeplearning device 1522, 1532, 1542, respectively (e.g., a CNN, RNN, otherdeep neural network, deep belief network, recurrent neural network,other machine learning, etc.) to aid in parameter selection,configuration, data processing, outcome determination, etc. While thedevices 1522, 1532, 1542 are depicted with respect to the engines 1430,1440, 1450 in the example of FIG. 15A, the devices 1522-1542 can beincorporated in the factories 1520-1540 as described above with respectto FIG. 8A, for example. The learning and improvement factories 1520,1530, 1540 implement a process of learning, feedback, and updating thedeployed deep learning devices 1522, 1532, 1542, for example. Theengines 1430, 1440, 1450 provide feedback to one or more of thefactories 1520-1540 to be processed and train an updated model to adjustsettings, adjust output, request input, etc. Periodically and/orotherwise upon reaching a threshold, satisfying a criterion, etc., thefactories 1520, 1530, 1540 can replace and/or re-deploy the deeplearning network model for the devices 1522, 1532, 1542, for example.

The deployed deep learning network (DLN) devices 1522, 1532, 1542 andassociated factories 1520, 1530, 1540 can be implemented using aprocessor and a memory particularly configured to implement a network,such as a deep learning convolutional neural network, similar to theexample networks 100, 200, 300 described above. Each factory 1520, 1530,1540 can be taught by establishing known inputs and outputs associatedwith an intended purpose of the network 1520, 1530, 1540. For example,the acquisition learning and improvement factory 1520 is tasked withimproving image acquisition settings for the acquisition engine 1430 toprovide to the imaging device 1410 based on patient information, reasonfor examination, imaging device 1410 data, etc. The reconstructionlearning and improvement factory 1530 is tasked with determining imagequality and reconstruction feedback based on acquired image data,imaging device 1410 settings, and historical data, for example. Thediagnosis learning and improvement factory 1540 is tasked with assistingin patient diagnosis based on patient information, image reconstructioninformation and analysis, and a clinical knowledge base, for example.

For each factory 1520, 1530, 1540, data sets are established fortraining, validation, and testing. A learning fraction to train andvalidate the factory 1520, 1530, 1540 and its included training networkmodel is a multiple of the validate and testing fraction of theavailable data, for example. The factory 1520, 1530, 1540 can beinitialized in a plurality of ways. For example, if no prior knowledgeexists about the component 1430, 1440, 1450 associated with therespective factory 1520, 1530, 1540, the training deep learning networkof the factory 1520, 1530, 1540 can be initialized using random numbersfor all layers except the final classifier layer of the network, whichcan be initialized to zero. If prior knowledge exists, network layers ofthe factory 1520, 1530, 1540 can be initialized by transferring thepreviously learned values to nodes in the network. Alternatively, evenwhen prior knowledge does not exist, network layers can be initializedusing a stacked auto-encoder technique.

In certain examples, feedback to and/or from the factory 1520, 1530,and/or 1540 is captured in storage (e.g., stored and/or buffered in acloud-based storage, etc.) including input data, actual output, anddesired output. When a sufficient amount of feedback is received, thetraining DLN of the corresponding factory 1520, 1530, 1540 is retrainedin an incremental fashion or newly trained using the additional feedbackdata (e.g., based on original feedback data plus additional feedbackdata, etc.) depending on the amount of feedback data received. Once(re)trained, the network model from the factory 1520-1540 can be used togenerate and/or re-deploy the deployed network model for the deeplearning device 1522-1542.

In certain examples, an auto-encoder technique provides unsupervisedlearning of efficient codings, such as in an artificial neural network.Using an auto-encoder technique, a representation or encoding can belearned for a set of data. Auto-encoding can be used to learn a model ofdata and/or other dimensionality reduction using an encoder and decoderto process the data to construct layers (including hidden layers) andconnections between layers to form the neural network.

For example, an auto-encoder can be implemented using a 3-layer neuralnetwork including an input layer, a hidden layer, and an output layer.In this example, the input layer and output layer include the samenumber of nodes or units but not all hidden layer nodes are connected toall nodes in the input layer. Rather, each node in the hidden layer isconnected to input nodes in a localized region of the input layer. Aswith the example of FIG. 3, the auto-encoder network can model portionsof an image to detect local patterns and/or features with a reducednumber of parameters, for example. The example auto-encoder can includetwo components: 1) an encoder function f that maps an input x to ahidden layer representation h=f(x), and 2) a decoder function g thatmaps h to reconstruct x for the output layer. Using weights and biases,the auto-encoder can be used to generate a new representation of theinput x through the hidden layer h.

Backpropagation or backward propagation of errors can be used in batches(e.g., mini-batches, etc.) involving pre-determined sets (e.g., smallsets) of randomly selected data from the learning data set usingstochastic gradient descent (SGD) to minimize or otherwise reduce apre-determined cost function while trying to prevent over-training byregularization (e.g., dropouts, batch normalization of mini-batchesprior to non-linearities, etc.) in the auto-encoder network. Usingmini-batches, rather than an entire training data set, the analysisshould converge more quickly. After leveraging an initial amount oftraining data to train the DLNs of the factories 1520, 1530, 1540 (e.g.,a multiple of validation data, etc.), for each subsequent batch of dataduring training, validation is performed and validation error ismonitored. Learning parameters with the best validation error aretracked and accumulated through the process to improve future training,for example. Parameters that provide the least error (e.g.,hyper-parameters) are selected after validation. Additionally, learningiterations can be stopped if the validation error does not improve aftera predetermined number of iterations. If validation error improves,iterations can continue until the validation error stabilizes. Then,parameters can be selected for the DLNs of the factories 1520, 1530,1540, for example.

Hyper parameters represent variables to be adjusted in the factories1520, 1530, 1540. In some examples, hyper parameters are selected for aparticular learning algorithm prior to applying that learning algorithmto the neural network. Hyper parameters can be fixed by hand and/ordetermined by algorithm, for example. In some examples, data used toselect hyper parameter values training data) cannot be used to test theDLNs of the factories 1520, 1530, 1540. Thus, a separate test data setis used to test the network once the hyper parameter values aredetermined using training data, for example.

Output and/or other feedback from the acquisition engine 1430,reconstruction engine 1440, and the diagnosis engine 1450 are providedto the system health module 1550 to generate an indication of the healthof the system 1500 based on an image quality indicator from thereconstruction engine 1440, a diagnosis confidence score provided by thediagnosis engine 1450, and/or other feedback generated by the deeplearning networks 1522, 1532, 1542 via the acquisition engine 1530,reconstruction engine 1440, and/or diagnosis engine 1450, for example.The output/feedback provided by the acquisition engine 1430,reconstruction engine 1440, and diagnosis engine 1450 to the systemhealth module 1550 is also provided to the learning and improvementfactories 1520, 1530 to update their network models based on the outputand/or other feedback. Thus, the learning and improvement factory 1520,1530 for a prior stage can be updated using feedback from a subsequentstage 1530, 1540, etc. The system health module 1550 can include its owndeployed deep learning device 1552 and system learning and improvementfactory 1555 for modeling and adjusting a determination of system healthand associated metric(s), recommendation(s), etc.

Deep Learning Networks identify patterns by learning the patterns.Learning includes tuning the parameters of the network using knowninputs and outputs. The learned network can predict the output given anew input. Thus, during the learning process, networks adjust theparameters in such a way to represent the mapping of genericinput-to-output mappings and, as a result, they can determine the outputwith very high accuracy.

Inputs to and outputs from the deployed deep learning device (DDLD)1522, 1532, 1542 can vary based on the purpose of the DDLD 1522, 1532,1542. For the acquisition DDLD 1522, for example, inputs and outputs caninclude patient parameters and imaging device 1410 scan parameters. Forthe reconstruction DDLD 1532, for example, inputs and outputs caninclude projection domain data and reconstructed data using acomputationally intensive algorithm. For the diagnosis DDLD 1542, inputcan include a two-dimensional and/or three-dimensional image, and outputcan include a marked visualization or a radiology report, for example. Atype of network used to implement the DDLD 1522, 1532, 1542 can varybased on target task(s). In certain examples, the correspondingacquisition, reconstruction, or diagnosis learning and improvementfactory 1520, 1530, 1540 can be trained by leveraging non-medical, aswell as medical, data, and the trained model is used to generate theDDLD 1522, 1532, 1542.

For example, the reconstruction engine 1440 provides feedback to theacquisition learning and improvement factory 1520, which can re-deploythe DDLD 1522 and/or otherwise update acquisition engine 1430 parametersbased on image quality and/or other output characteristics determined bythe reconstruction engine 1440. Such feedback can be used by theacquisition engine 1430 to adjust its settings when modeled andprocessed by the DDLD 1522. The reconstruction engine 1440 can alsoprovide feedback to its own reconstruction learning and improvementfactory 1530. The acquisition engine 1430 can also provide feedback toits own acquisition learning and improvement factory 1520.

Similarly, for example, the diagnosis engine 1450 provides feedback tothe learning and improvement factory 1530 for the reconstruction engine1440, which can re-deploy the DDLD 1532 and/or otherwise updatereconstruction engine 1440 parameters based on a confidence scoreassociated with diagnosis and/or other output characteristics determinedby the diagnosis engine 1450. Such feedback can be used by thereconstruction engine 1440 to adjust its settings when modeled andprocessed by the DDLD 1532.

One or more of the learning and improvement factories 1520, 1530, 1540can also receive feedback from one or more human users 1404 (e.g., basedon using the outcome of the diagnosis engine 1450 to diagnose and treatthe patient 1406, etc.). By chaining feedback between engine(s)1430-1450, factories 1520-1540 d the system health module 1550, engines1430, 1440, 1450 can learn and improve from the current and/orsubsequent phase of the imaging and diagnosis process.

Thus, certain examples consider a reason for examination of a patient inconjunction with an acquisition deployed deep learning device 1522, areconstruction deployed deep learning device 1532, a diagnosis deployeddeep learning device 1542, and a system health deployed deep learningdevice 1552 to improve configuration and operation of the system 1500and its components such as the imaging device 1410, informationsubsystem 1420, acquisition engine 1430, reconstruction engine 1440,diagnostic engine 1450, etc. Deep learning can be used for imageanalysis, image quality (e.g., quality of clarity, resolution, and/orother image quality feature, etc.), etc.

A learning data set can be applied as input to each learning andimprovement factory 1520, 1530, 1540. The learning data set can includean image data set with assigned image quality metric (e.g., a scale of1-5, etc.) as an output, for example. The system 1500 and its componentsevaluate one or more metrics as outputs and feedback to the factories1520, 1530, 1540 for continued improvement. Automating inputs andoutputs to the factories 1520, 1530, 1540, 1555, as well as the DDLDs1522, 1532, 1542, 1552, facilitates continued system operation andimprovement.

In certain examples, using a 3D topography of medical images fromdifferent imaging modalities (MRI, CT, x-ray, etc.) can provide changesin classification, convolution, etc. A model can be formed by therespective DDLD 1522, 1532, 1542. The model(s) can be adjusted based onanatomy, clinical application, patient information (e.g., data and/orscout scan, etc.), patient history, etc.

In certain examples, each DDLD 1522, 1532, 1542 determines a signature.For example, the DDLD 1522, 1532, 1542 determines signature(s) formachine (e.g., imaging device 1410, information subsystem 1420, etc.)service issues, clinical issues related to patient health, noise textureissues, artifact issues, etc. The DDLD 1522, 1532, 1542 can determine asignature indicative of one of these issues based on input, learnedhistorical patterns, patient history, preference, etc.

Certain examples provide metrics for validation and regressive testingvia the DDLD 1522, 1532, 1542. Output can also include notice(s) fromsignature classification(s). Certain examples provide an image qualitymatrix/metrics for human visual inspection. Certain examples provide animage quality matrix/metrics for non-human interpretation (e.g., bigdata analytics, machine learning, etc.).

Certain examples can provide an output for quality control (e.g.,provide a number or value to reflect an overall quality of an imagingscan, etc.). Certain examples provide an output for rescan assistance(e.g., deciding whether a rescan is warranted, etc.). Certain examplescan be used to automate protocol selection and/or new protocolcustomization (e.g., new protocol parameters can be computed based on animage quality metric, etc.).

In certain examples, the output can be used to improve development ofhardware systems. For example, if an issue is identified in a medicalsystem (e.g., an artifact caused by hardware, etc.), a next iterationcan propose a solution to fix or alleviate the issue. In certainexamples, clinical context is added to the DDLD 1522, 1532, 1542 tofacilitate clinical decision making and support.

Output from the DDLD 1522, 1532, 1542 can be used to improve developmentof algorithms such as algorithms used to measure quality. By providingfeedback to the factories 1520, 1530, 1540, a change to an algorithm canbe modeled and tested via a DLN of the respective factory 1520, 1530,1540 to determine how the change can impact output data. Capturing andmodeling changes in a feedback loop can be used with nonlinear,iterative reconstruction of acquired image data, for example.

Certain examples facilitate monitoring and adjustment of machine healthvia automated diagnosis by the system 1500. Service decisions can bemade (e.g., an automated service that the machine can run on itself, acall for manual human repair, etc.) based on deep learning outputinformation. Machine-based decision support can be provided, and one ormore machine-specific signatures indicative of an issue can beinvestigated and adjusted.

Certain examples can extrapolate additional information about a patientbased on patient information input from the information subsystem 1420combined with output from the acquisition engine 1430, reconstructionengine 1440, and diagnosis engine 1450 in conjunction with their DDLDs522, 1532, 1542. In certain examples, based on patient history, medicalissue, past data sets, etc., the DDLD 1522 can help determine whichacquisition settings are best to acquire an image data set, and the DDLD1532 can help determine what protocol is the best selection to provideimage data set output. Patient behavior, such as movement during scans,how their body handles contrast, the timing of the scan, perceived dose,etc., can be gathered as input by the DDLD 1522, for example, todetermine image device 1410 acquisition settings, for example.

Certain examples provide an end-to-end image acquisition and analysissystem including an improved infrastructure chaining multiple DDLDs1522, 1532, 1542 together. For example, raw data acquired by the imagingdevice 1410 and acquisition engine 1430 is processed and provided to thereconstruction engine 1440 to produce one or both of a) amachine-readable image provided to the diagnostic decision supportengine 1450 and b) a human-viewable image displayed for user diagnosis.Different DLNs are provided for acquisition, reconstruction, anddiagnosis, and each DDLD 1522, 1532, 1542 has different input, differentprocessing, and different output.

Thus, the example system 1500 creates one or more images usinginterconnected DDLDs 1522, 1532, 1542 and corresponding engines 1430,1440, and links the image(s) to decision support via the diagnosisengine 1450 and DDLD 1542 for diagnosis. Real-time (or substantiallyreal time given processing and transmission delay) feedback (e.g., feedforward and feed back between learning and improvement factories 1520,1530, 1540 and engines 1430, 1440, 1450) loops are formed in the examplesystem 1500 between acquisition and reconstruction and between diagnosisand reconstruction, for example, for ongoing improvement of settings andoperation of the acquisition engine 1430, reconstruction engine 1440,and diagnosis engine 1450 (e.g., directly and/or by replacing/updatingthe DDLD 1522, 1532, 1542 based on an updated/retrained DLN, etc.). Asthe system 1500 learns from the operation of its components, the system1500 can improve its function. The user 1404 can also provide offlinefeedback (e.g., to the factory 1520, 1530, 1540, etc.). As a result,each factory 1520, 1530, 1540 learns differently based on system 1500input as well as user input in conjunction with personalized variablesassociated with the patient 1406, for example.

In certain examples, the diagnosis engine 1450 operates with the DDLD1542, which is trained, validated and tested using sufficiently largedatasets that can adequately represent variability in the expected datathat the diagnosis engine 1450 is to encounter. The diagnosis learningand improvement factory 1540 can be used to refine its output as moreinput is provided to it by the diagnostic engine 1450 and/orreconstruction engine 1440, for example. The factory 1540 can thenreplace the deployed DLN of the DDLD 1542, for example.

The diagnosis engine 1450 identifies pattern(s) in one or more imagesbased on big data from patients in a population (e.g., retrieved fromthe information subsystem 1420) to suggest a diagnosis of the patient1406 to the user 1404. The example diagnosis engine 1450 highlightsarea(s) for user 1404 focus and can predict future area(s) of interestbased on big data analytics, for example. Even if the image is presentedin a suboptimal way, the diagnosis engine 1450 can provide apatient-dependent answer, rather than a determination one dependent onthat particular imaging scan. The diagnosis engine 1450 can analyze theimage and identify trouble spot(s) that the user 1404 may not see basedon settings used in acquisition, reconstruction, analysis, etc. Outputcan be automatic to trigger another system/device and/or can bepresented as a suggestion to the user 1404. Data output from the system1500 can be provided to a cloud-based system, for example. Output can beprovided to the system learning and improvement factory 1555 of thesystem health module 1550 such that the system health module 1550 learnswhen actions should be taken to maintain or improve health of the system1500.

The system health module 1550 receives input from a plurality ofcomponents 1430, 1440, 1450 and processes the input to determine whetherchanges should be made to the system 1500. Based on exposure to andlearning from issues affecting the acquisition engine 1430,reconstruction engine 1440, diagnosis engine 1450, etc., the systemhealth module 1550 provides an output to the acquisition engine 1430 tomodify behavior of the imaging device 1410 and/or other systemcomponent. The system health module 1550 also provides an output for thesystem design engine 1560, which uses the identified problem/issue tomodify design of the imaging device 1410 and/or system 1400, 1500component, for example.

FIG. 15B illustrates an example system implementation 1501 in which theacquisition engine 1430, reconstruction engine 1440, and diagnosisengine 1450 are accompanied by a data quality assessment engine 1570, animage quality assessment engine 1572, and a diagnosis assessment engine1574. In the configuration 1501 of FIG. 15B, each engine 1430, 1440,1450 receives direct feedback from an associated assessment engine 1570,1572, 1574. In certain examples, the acquisition engine 1430,reconstruction engine 1440, and/or diagnosis engine 1450 receivesfeedback without having to update their associated deployed deeplearning modules 1522, 1532, 1542. Alternatively or in addition, thedata quality assessment engine 1570, image quality assessment engine1572, and diagnosis assessment engine 1574 provide feedback to theengines 1430-1450. Although direct connections are depicted in theexample of FIG. 15B for the sake of simplicity, it should be understoodthat each of the deep learning-based feedback modules 1570-1574 has anassociated training image database including different classes ofexample conditions, an associated learning and improvement factorymodule, an orchestration module, and a trigger for associated parameterupdate and restart, for example.

Thus, the acquisition engine 1430 may receive feedback from the dataquality assessment engine (DQ-AE) 1570, the image quality assessmentengine (IQ-AE) 1572, and/or the diagnosis assessment engine (Diag-AE)1574, for example. The reconstruction engine 1440 may receive feedbackfrom the IQ-AE 1572 and/or the Diag-AE 1574, for example. The diagnosisengine 1450 may receive feedback from the Diag-AE 1574, for example.

FIG. 16 illustrates a flow diagram of an example method 1600 forimproved image acquisition, processing, and patient diagnosis. At block1602, personalized patient characteristics are input into theacquisition engine 1430. For example, a clinician may enter personalizedpatient characteristics into the acquisition engine 1430. Alternativelyor in addition, personalized patient characteristics can be provided foran identified patient to be image via the information subsystem 1420. Atblock 1604, the acquisition engine 1430 suggests one or more imagingdevice 1410 settings based on the input personalized patientcharacteristics as well as learned information extracted from the DDLD1522, for example.

Thus, using information particular to the patient 1406 as well asinformation learned by the DDLD 1522, improved settings for imageacquisition by the imaging device 1410 can be determined. At block 1606,one or more images of the patient 1406 are obtained by the imagingdevice 1410. The images are obtained according to the settings providedby the acquisition engine 1430. The settings can be automaticallyconfigured at the imaging device 1410 by the acquisition engine 1430and/or manually input/overridden by the user 1404 (e.g., a clinician,radiologist, technician, etc.).

At block 1608, the reconstruction engine 1440 receives raw image datafrom the acquisition engine 1430 and processes the image data to assignimage quality metric(s). The image quality (IQ) metric can be acomprehensive image quality indicator (IQI) and/or one or moreparticular metrics regarding aspects of image quality. For example,specific image quality metrics include spatial resolution, noise, etc.At block 840, described above, feedback generated by the reconstructionengine 1440 can be collected and stored. Thus, lessons learned by thesystem 1500 from the reconstruction of the acquired image data can befed back into the acquisition learning and improvement factory 1520 forfurther refinement of imaging device 1410 settings. After conducting animage quality analysis on the image data, the reconstruction engine 1440processes the image data to reconstruct an image for further review andanalysis. This resulting image or images can be processed for automatedmachine analysis, such as computer-aided diagnosis (CAD), or for humanviewing of the image.

Configuration settings from the reconstruction DDLD 1532 is used todetermine whether the acquired image data is to be processed for machineanalysis and/or human viewing. At block 1612, the image is reconstructedfor human review the display of the resulting image. At block 1614, theimage data is processed to produce an image suitable for machineevaluation and analysis of the image. With the machine-analyzable image,for example, features of the image can be optimized for computerdetection but need not be visually appreciable to a user, such as aradiologist. For the human-viewable image, however, features of theimage anatomy should be detectable by a human viewer in order for thereconstruction to be useful.

At block 1616, if the human viewable image has been reconstructed, thereconstruction agent 1440 provides the image to the diagnosis engine1450 which displays the image to the user 1404. At block 1618, themachine analyzable image has been generated, then the reconstructionengine 1440 provides the machine-readable image to the diagnosis engine1450 automated processing and suggested diagnosis based on the imagedata from the diagnosis engine 1450.

At block 840, feedback regarding the human-viewable image and/ormachine-suggested diagnosis is provided from the diagnosis engine 450.At block 1624, a diagnosis of the patient 1406 is made based on humanviewing of the image by the user 1404 and/or automated processing of theimage by the diagnosis engine 1450, taken alone or in conjunction withdata from the DDLD 1542 and/or information subsystem 1420.

The diagnosis can be provided to the user 1404, the patient 1406, and/orrouted to another system, for example. For example, at block 840,feedback is provided from the diagnosis engine 1450 and/or the user1404. Feedback can also be provided to the system design engine 1560.Feedback from the user 1404, diagnosis engine 1450, reconstructionengine 440, acquisition engine 1430, and/or other system 1500 componentcan be provided to the system health module 1560 to compute anindication of system 1500 health.

FIG. 17 illustrates an example data flow and transformation ofinformation 1700 as it flows among the components of the system 1500. Asshown in the example of FIG. 17, a first data 1702 is provided by theimaging device 1410 to the acquisition engine 1430. A second data 1704is provided by the information subsystem 1420 to the acquisition engine1430. The acquisition engine 1430 sends third data 1706 including thefirst data 1702 and/or second data 1704 to the acquisition DDLD 1522.The acquisition DDLD 1522 transforms the third data 1706 into fourthdata 1708, and sends the fourth data 1708 back to the acquisition engine1430. The acquisition engine 1430 sends the fourth data 1708 to theimaging device 1410. The acquisition engine 1430 sends fifth data 1710to the reconstruction engine 1440.

The reconstruction engine 1440 sends sixth data 1712 including the fifthdata 1710 to the reconstruction DDLD 1532. The DDLD 1532 transforms thesixth data 1012 into seventh data 1714, and sends the seventh data 1714back to the reconstruction engine 1440. The reconstruction engine 1440sends the seventh data 1714 to the acquisition engine 1430. Thereconstruction engine 1440 sends eighth data 1716 to the diagnosisengine 1450.

The diagnosis engine 1450 sends ninth data 1718 including the eighthdata 1716 to the diagnosis DDLD 1542. The DDLD 1542 transforms the ninthdata 1718 into tenth data 1720, and sends the tenth data 1720 back tothe diagnosis engine 1450. The diagnosis engine 1450 sends the tenthdata 1720 to the reconstruction engine 1440.

Thus, certain examples transform patient information, reason forexamination, and patient image data into diagnosis and otherhealthcare-related information. Using machine learning, such as deeplearning networks, etc., a plurality of parameters, settings, etc., canbe developed, monitored, and refined through operation of imaging,information, and analysis equipment, for example. Using deep learningnetworks, for example, learning/training and testing can be facilitatedbefore the imaging system is deployed (e.g., in an internal or testingenvironment), while continued adjustment of parameters occurs “in thefield” after the system has been deployed and activated for use, forexample.

Certain examples provide core processing ability organized into units ormodules that can be deployed in a variety of locations. Off-deviceprocessing can be leveraged to provide a micro-cloud, mini-cloud, and/ora global cloud, etc. For example, the micro-cloud provides a one-to-oneconfiguration with an imaging device console targeted for ultra-lowlatency processing (e.g., stroke, etc.) for customers that do not havecloud connectivity, etc. The mini-cloud is deployed on a customernetwork, etc., targeted for low-latency processing for customers whoprefer to keep their data in-house, for example. The global cloud isdeployed across customer organizations for high-performance computingand management of information technology infrastructure with operationalexcellence.

Using the off-device processing engine(s) (e.g., the acquisition engine1430, reconstruction engine 1440, diagnosis engine 1450, etc., and theirassociated deployed deep learning network devices 1522, 1532, and/or1542, etc.), acquisition settings can be determined and sent to theimaging device 1410, for example. For example, purpose for exam,electronic medical record information, heart rate and/or heart ratevariability, blood pressure, weight, visual assessment of prone/supine,head first or feet first, etc., can be used to determine one or moreacquisition settings such as default field of view (DFOV), center,pitch, orientation, contrast injection rate, contrast injection timing,voltage, current, etc., thereby providing a “one-click” imaging device.Similarly, kernel information, slice thickness, slice interval, etc.,can be used to determine one or more reconstruction parameters includingimage quality feedback, for example. Acquisition feedback,reconstruction feedback, etc., can be provided to the system designengine 1560 to provide real-time (or substantially real-time givenprocessing and/or transmission delay) health analytics for the imagingdevice 1410 as represented by one or more digital models (e.g., deeplearning models, machine models, digital twin, etc.). The digitalmodel(s) can be used to predict component health for the imaging device1410 in real-time (or substantially real time given a processing and/ortransmission delay).

Each deep learning network can be trained using curated data withassociated outcome results. For example, data regarding stroke (e.g.,data from onset to 90 days post-treatment, etc.) can be used to train aneural network to drive to predictive stroke outcomes. Thus,operational, clinical, treatment, and therapy “biomarkers” can beidentified for best and/or other improved outcomes. Similarly, lungcancer data can be analyzed by a deep learning network includingdepartment to department tracking from screening, diagnosis, treatmentplanning, treatment response, final outcome, etc., for one or moreimaging devices 1410 including CT, PET/CT, nuclear medicine, etc.

For image acquisition, given one or more known inputs and one or moreknown outputs, acquisition settings can be determined automatically totrain the acquisition learning and improvement factory 1520 forpredictable output to generate the deployed DDLD 1522. When the outputof settings for the imaging device 1410 reaches a threshold ofreliability, the acquisition engine 1430 can be certified to provideacquisition settings for the imaging device 1410 (e.g., as integratedinto the imaging device 1410 and/or as a separate device incommunication with the imaging device 1410, etc.). Settings can be usedand modified without further customer training or testing. As a result,a user can obtain high quality image acquisitions and avoid a bad orsubpar set of image acquisitions. Acquisition settings and theassociated DDLD 1522 can be trained to respond only to good qualityimage acquisitions, and setting adjustments can be suggested by the DDLD1522 when a bad quality image is obtained. Thus, from a userperspective, one button is pushed to consistently acquire a fast imageexam. Using a purpose for the examination in conjunction with patientparameters, a DICOM header for a desired output, and an indication ofthe desired output obtained from an existing medical exam whichcorresponds to imaging device 1410 parameters for successful and/orunsuccessful data sets, the DDLD 1522 can recognize good image qualityand suggest corresponding settings as default settings, as well as, whenuser makes a mistake in configuring the imaging device 1410 for imageacquisition, suggest settings to recover from the mistake. Over time,the acquisition learning and improvement factory 1520 can evolve andimprove based on learned successes and failures to re-train andre-deploy an improved DDLD 1522 to drive to the acquisition engine 1430,for example.

In certain examples, cloud-based protocols can be captured and managedto automate selection of protocol and/or rules to make best practicesavailable via the cloud.

Quality feedback can also be obtained from image reconstruction. Withouthuman review, a good or bad image can be identified and associated withone or more image quality metrics and/or indicators by the DDLD 1532,for example. Such an image quality index (IQI) and/or other metric canbe generated by the reconstruction engine 1440 using DDLD 1532 and usedto make a medical decision with or without human review, for example.The generated index/metric can be used to inform the DDLD 1532 and/oruser 1404 regarding whether the imaging device 1410 is acquiring good orbad quality images and under what conditions, etc.

In deep learning, testing can assess quality of images automatically forknown cases at a certain level. Feedback based on an analysis of imagequality compared to imaging device 1410 settings can be provided to thesystem design engine 1560 to facilitate further imaging device 1410development, for example. Using the system design engine 1560 andlearning and improvement factories 1520 and/or 1530, a decline in imagequality can be detected and used to evaluate system health 1550,including health of the imaging device 1410, for example. While a humanuser 1404 may not detect a gradual decrease in quality, deep learningprovides an objective, unbiased evaluation for early detection.

In certain examples, a diagnostic index or detectability index can becalculated similar to an image quality indicator. The diagnostic indexcan be a measure of under what conditions the user 1404 can make adiagnosis given a set of data. The DDLD 1542 and associated diagnosislearning and improvement factory 1540 analyze current and historicaldata and system 1500 parameters from other components to provide aconsistent indication for the user 1404, patient 1406, type ofcondition, type of patient, type of examination, etc. Once the trainingDLN of the factory 1540 is trained, for example, the model can bedeployed to the DDLD 1542, and diagnosis data can be compared to imagequality. Feedback can be provided to the reconstruction engine 1440,acquisition engine 1430, the associated learning and improvementfactories 1520, 1530, and/or the user 1404 to provide further indicationof image quality and/or a corresponding change in imaging device 1410settings for acquisition, for example.

In some examples, instead of or in addition to a numerical indication ofpatient diagnosis, a holistic analysis/display can be provided. Using aholistic analysis, visual indication, such as a heat map, deviation map,etc., can be provided to visualize how the patient 1406 fits or does notfit with trends, characteristics, indicators, etc., for a particulardisease or condition. In certain examples, as the factory 1540 improvesin its diagnosis learning, the visual representation can improve. Usinga holistic approach with the diagnosis engine 1450 and its DDLD 1542,data from a plurality of sources is processed and transformed into aform in which a human can identify a pattern. Using deep learning, theDDLD 1542 can process thousands of views of the data, where a human user1404 may only be able to reasonably process ten before losing focus.

The deep learning process of the DDLD 1542 can identify pattern(s) (andpotentially enable display an indication of an identified pattern viathe diagnosis engine 1450) rather than the human user 1404 having tomanually detect and appreciate (e.g., see) the pattern. Multi-variantanalysis and pattern identification can be facilitated by the DDLD 1542,where it may be difficult for the human user 1404 to do so. For example,the DDLD 1542 and diagnosis engine 1540 may be able to identify adifferent pattern not understandable to humans, and/or a pattern that isunderstandable to humans but buried in too many possibilities for ahuman to reasonably review and analysis. The DDLD 1542 and diagnosisengine 1450 can provide feedback in conjunction with human review, forexample.

In certain examples, the holistic analysis feeds into the diagnosis madeby the user 1404, alone or in conjunction with the diagnosis engine1450. The diagnosis engine 1450 and its DDLD 1542 can be used to providea second opinion for a human decision as a legally regulated/medicaldevice, for example. In certain examples, the diagnosis engine 1450 canwork in conjunction with the DDLD 1542 to provide automated diagnosis.

Example Analytics Framework

In certain examples, a healthcare analytics framework 1800 can beprovided for image acquisition, image reconstruction, image analysis,and patient diagnosis using the example systems 1400, 1500 (includingthe acquisition engine 1430, reconstruction engine 1440, and diagnosisengine 1450, along with their associated DDLDs 1520-150 and learning andimprovement factories 1520-1540). As shown in the example of FIG. 18,input, such as imaging device 1410 parameter, reconstruction engine 1440parameter, etc., is provided to a physics-based device comparison 1810.The device comparison 1810 can be facilitated by the DDLD 1522 and/orother machine learning network, for example. The input is used by thedevice comparison 1810 to compare the imaging device 1410 to otherimaging devices of the same and/or different types from the same and/ordifferent vendor. A deep learning and/or other machine learningtechnique can be used to learn and assist in developing the comparisonbased on device 1410 characteristics, image acquisition parameter,reconstruction setting, etc. For example, the DDLD 1522 and/or 1532 canbe used for the device comparison 1810 to develop a profile and/or othercomparison of the imaging device 1410 to other imaging scanners.

The device 1410 comparison information is provided to a data evaluationspecification 1820. The data evaluation specification 1820 can also beimplemented by the DDLD 1522 and/or 1532 and/or a separate processor,for example. The data evaluation specification 1820 processes areconstructed image 1822 from the reconstruction engine 1440 and atransform of raw image data 1824 from the acquisition engine 1430 andthe imaging device 1410. Machine learning methods such as deep learning,dictionary learning (e.g., build a dictionary from other images andapply the dictionary definitions to the current image, etc.), etc., canbe applied to the reconstructed and/or raw image data to define imageattributes and/or task-based image quality evaluation metrics 1826.Image quality information (e.g., noise, resolution, etc.) can beextracted directly from the image and raw image data (e.g., with regionof interest, without using specific phantoms and/or modulation transferfunction (MTF), etc.) using deep learning and/or other machine learningtechnique. Additionally, one or more task-based metrics (e.g.,detectability, etc.) can be extracted from the data using deep learningand/or other machine learning. The attribute(s) and metric(s) 1826 forma specification for data evaluation based on the device-basedcomparison.

In certain examples, a model can be formed. The data evaluationspecification 1820 constructs a transfer function 1828 to mathematicallyrepresent or model inputs to and outputs from the data evaluationspecification 1820. The transfer function 1828 helps to generate andmodel the image attributes and/or task-based image quality evaluationmetrics 1826. In certain examples, variation can be modeled based onanalytics such calculating a nodule volume, estimating a source ofvariations from image(s) directly, etc., and modeled variation can beused to standardize the reconstructed image and improve analytics.

Based on the model and transfer function 1828 providing analytics andmodification of image data, an outcomes processor 1830 determines one ormore clinical outcomes. For example, information can be provided to theoutcomes processor 1830 to facilitate (e.g., via the diagnosis engine1450) user determination of a clinical outcome. Alternatively or inaddition, the outcomes processor 1830 can generate a machinedetermination (e.g., using the diagnosis engine 1450 and DDLD 1542 withimage analysis) of clinical outcome.

Thus, for example, image resolution quality has traditionally beenmeasured using a phantom(s) (e.g., a wire, edge, etc.) in conjunctionwith MTF. However, many radiologists can tell that a clinical image hasa lower resolution by observing the image. The deep learning and/orother machine network learns to mimic this observation through repeatedexposure and analysis, for example. For example, using information fromthe device comparison 1810 in conjunction with the reconstructed image1822, standardization transform 1824, etc., the data evaluationspecification 1820 can enable the reconstruction engine 1440 and itsDDLD 1532, for example, to compute image attributes and recalibratetransformation to work with the diagnosis engine 1450 and its DDLD 1542to provide analytics to clinicians and to identify acceptable orunacceptable resolution in the image with respect to a range, threshold,etc., that is defined and/or learned by the DDLD 1542, for example. Ifthe resolution is unacceptable, then the DDLD 1522 can be updated viathe learning and improvement factory 1520, and acquisition engine 1430settings are adjusted, for example.

Image Acquisition Examples

FIG. 19 illustrates a flow diagram of an example method 1900 for imageacquisition using the imaging device 1410 and the image acquisitionengine 1430. At block 1902, personalize patient variables are input tothe acquisition engine 1430. Personalized patient variables can includepatient height, patient weight, imaging type, reason for exam, patienthealth history, etc. For example, a clinician may enter personalizedpatient characteristics into the acquisition engine 1430. Alternativelyor in addition, personalized patient characteristics can be provided foran identified patient to be image via the information subsystem 1420.

At block 1904, the acquisition deployed deep learning network device1522 analyzes the input to the acquisition engine 1430. For example, theDDLD 1522 processes patient parameters, prior imaging device 1410 scanparameters, etc., to generate imaging device 1410 settings for imageacquisition. Using a CNN, RNN, autoencoder network, and/or otherdeep/machine learning network, the DLN 520 leverages prior acquisitionsin comparison to current imaging device 1410 settings, patientinformation, reason for exam, patient history, and population healthinformation, etc., to generate a predictive output. Relationshipsbetween settings, events, and results can be explored to determineappropriate imaging device 1410 settings, ideal or preferred acquisitionsettings based on type of exam and type of patient, changes to imagingdevice 1410 design, etc. Settings can include intensity or radiationdosage settings for sufficient (versus poor and/or versus high quality,etc.) image quality, etc. Settings can include acquisition type,duration, angle, number of scans, position, etc.

At block 1906, the acquisition engine 1430 suggests one or more imagingdevice 1410 settings based on the input personalized patientcharacteristics as well as learned information extracted from the DDLD1522, for example. As described above, output from the DDLD 1522 can beorganized as one or more parameters or configuration settings for theimaging device 1410 to obtain images of the patient 1406. Thus, usinginformation particular to the patient 1406 as well as informationlearned by the deployed deep learning network device 1522, improvedsettings for image acquisition by the imaging device 1410 can bedetermined. Based on the reason for exam, particular patientinformation, and existing imaging device 1410 settings, the acquisitionDDLD 1522 can generate suggested settings for use by the acquisitionengine 1430 to obtain image data from the patient 1406 via the imagingdevice 1410. The settings can be automatically applied by theacquisition engine 1430 to the imaging device 1410 and/or manuallyentered/overridden by the user 1404, for example.

At block 1908, one or more images of the patient 1406 are obtained bythe imaging device 1410. The images are obtained according to thesettings provided by the acquisition engine 1430. The settings can beautomatically configured at the imaging device 1410 by the acquisitionengine 1430 and/or manually input/overridden by the user 1404 (e.g., aclinician, radiologist, technician, etc.), for example.

At block 1910, acquired image data is sent to the reconstruction engine1440 (e.g., to be reconstructed into a human-viewable image and/ormachine-processed, etc.). The reconstruction engine 1440 (using the DDLD1532) can generate an image quality (IQ) metric to be a comprehensiveimage quality indicator (IQI) and/or one or more particular metricsregarding aspects of image quality associated with the acquired rawimage data. For example, specific image quality metrics include spatialresolution, noise, etc.

At block 1912, feedback from the reconstruction engine 1440 is providedto the acquisition learning and improvement factory 1520 to improve theDDLD 520 (e.g., generate a new DDLD 1522 for deployment with theacquisition engine 1430) for imaging device 1410 settings generation andrecommendation. Thus, lessons learned by the system 1500 from thereconstruction of the acquired image data can be fed back into theacquisition learning and improvement factory 1520 (and/or the imagequality assessment engine 1572, etc.) for further refinement of networkoperation and resulting improvement in imaging device 1410 settings. Thefeedback ensures an ongoing improvement to the DDLD 1522 (via thefactory 1520) and, as a result, to the settings provided to the imagingdevice 1410 for image acquisition for various patients 1406.

At block 1914, if the image is not displayed, then no additionalfeedback is obtained. However, if the image is displayed, then, at block1916, additional feedback is provided to the acquisition learning andimprovement factory 1520. For example, one or more of the reconstructionDDLD 1532, diagnosis engine 1450, diagnosis DDLD 1542, user 1404, etc.,can provide further feedback to the acquisition learning and improvementfactory 1520 (and/or the image quality assessment engine 1572, etc.) toimprove its learning and data processing. For example, feedbackregarding kernel used, noise reduction setting, slice thickness,interval, etc., can be provided.

In certain examples, the acquisition engine 1430 and associated DDLD1522 can be implemented as a device that can be connected to the imagingdevice 1410 to configure the operation of the imaging device 1410 forimage acquisition. The acquisition configuration device can be used by atechnician or installer on-site, sold to a customer for their ownoperation of the device, etc.

FIG. 20 illustrates example image acquisition configuration system 2000including a training learning device 2010 and an image acquisitionconfiguration device 2020 communicating with the imaging system 1401.The image acquisition configuration device 2020 includes a processor2030 and a memory 2040. The processor 2030 of the device 2020instantiates a deployed deep learning device 2050.

In certain examples, the image acquisition configuration device 2020 isinitially configured using a first set of imaging system configurationparameters determined by training and testing a prior set of parametersusing the training learning device 2010. The device 2020 also includesits own deployed learning device 2050 to be generated using the traininglearning device 2010 to assist the image acquisition configurationdevice in determining configuration parameters based on systemoperation, patient input, and/or etc. The device 2050 operates with theconfiguration device 2020 while the training device 2010 receivesfeedback and continues to evolve. The training device 2010 can re-deploythe deployed learning device 2050 when certain criterion is satisfied(e.g., threshold of feedback collected, margin of deviation betweendevice 2050 outcome and modeled device 2010 outcome, etc.). The devices2010, 2050 can be implemented such as described above, for example.

The device 2020 and its initial parameters can be deployed using thetraining learning device 2010 once the network of the device 2010 hasbeen trained (e.g., reaches a reliability threshold associated withimaging device 1410 configuration parameters for a type of patient, typeof examination, type of image acquisition, image quality threshold,etc.). Until the configuration device 2020 is generating relevant,reliable settings recommendations for the imaging device 1410, thetraining device 2010 continues to provide recommendations andadjustments and incorporates feedback on inefficiencies, inaccuracies,etc., to continue to learn and refine parameter selection for a givenexamination purpose, patient type, condition type, dosage, safetythreshold, operating condition, etc., and to redeploy the deployedlearning device 2050.

Once the learning device 2050 has been deployed, and the configurationdevice 2020 has been validated as having sufficient reliability andquality, the image acquisition configuration device 2020 can be releasedfor deployment and installation at a customer site (e.g., for use by atechnician associated with the imaging device 1410 manufacturer, by thecustomer directly, etc.). The device 2020 can be deployed to physicalconnect or “plug in” to the imaging device 1410 (e.g., by wire, cable,connector, etc. Alternatively or in addition, the image acquisitionconfiguration device 2020 can communicate wirelessly (e.g., viaBluetooth™ Bluetooth Low Energy (BLE™), near field communication (NFC),Wi-Fi™, etc.) with the imaging device 1410 to transmit settings to theimaging device 1410 and receive feedback from the imaging device 1410.Feedback is used by the training learning device 2010 to continue tolearn, modify, and improve setting recommendations, for example, duringoperation of the devices 1410, 2020. The device 2020 can alsocommunicate with the reconstruction engine 1440 and/or other imageprocessing workstation to obtain image quality feedback for imagesresulting from image data acquisition by the imaging device 1410. Thetraining device 2010 uses this feedback as well to train further torespond to good quality and to make suggestions for bad quality imageacquisition, for example.

FIG. 21 illustrates a flow diagram of an example method 2100 to trainand deploy the image acquisition configuration device 2020. At block2102, the image acquisition configuration device 2020 is trained usingthe training device 2010. For example, the device 2020 and its initialconfiguration settings can be trained first using the training learningdevice 2010 based on training and testing one or more reference sets ofparameters by the device 2010.

At block 2104, training continues until an output of the traininglearning device 2010 satisfies a threshold (e.g., compared to a knownresponse from the reference data. Both good reference data and badreference data can be used to train the device 2010 to recognize usableoutcomes and unusable outcomes, for example. Until the training device2010 reaches a reliability threshold associated with imaging device 1410configuration parameters for a type of patient, type of examination,type of image acquisition, image quality threshold, etc., the trainingdevice 2010 continues to train and incorporates feedback oninefficiencies, inaccuracies, etc., to continue to learn and refineparameter selection for a given examination purpose, patient type,condition type, dosage, safety threshold, operating condition, etc.

Once the training device 2010 has been trained and validated as havingsufficient reliability and quality, then, at block 2106, the deployedlearning device 2050 is generated. For example, the trained network ofthe training device 2010 is used to generate a deployment network modelfor the deployed learning device 2050.

At block 2108, the image acquisition configuration device 2020 can bereleased for deployment at a customer site (e.g., for use by atechnician associated with the imaging device 1 manufacturer, by thecustomer directly, etc.). The device 2020 can be deployed to physicalconnect or “plug in” to the imaging device 1410 (e.g., by wire, cable,connector, etc. Alternatively or in addition, the image acquisitionconfiguration device 1120 can communicate wirelessly (e.g., viaBluetooth™, Bluetooth Low Energy (BLE™), near field communication (NFC),Wi-Fi™, etc.) with the imaging device 1410 to transmit settings to theimaging device 1410 and receive feedback from the imaging device 1410.

At block 2110, the image acquisition configuration device 2020 cangenerate configuration parameters for the imaging device 1410 to obtainone or more images of the patient 1406. Based on the patient 1406, priorpatient information (e.g., patient medical history, family history fromthe information subsystem 1420, two-dimensional and/or three-dimensionalscout scans, etc.), reason for and/or type of examination, etc., thedeployed network 2050 of the configuration device 2020 generatesconfiguration parameters to configure the imaging device 1410 for imageacquisition of the patient 1406.

At block 2112, configuration parameters are output by the configurationdevice 2020. The configuration device 2020 can provide the parameters tothe imaging device 1410 to automatically configure the imaging device1410 for image acquisition (with operator override and/or adjustment,for example). The configuration device 2020 can also provide theparameters to the operator to be used to set up the imaging device 1410for image acquisition.

At block 2114, feedback is received from the imaging device 1410 by theimage acquisition configuration device 2020. For example, the feedbackcan be received via wired and/or wireless connection during and/or afterimage acquisition by the imaging device 1410. In certain examples,feedback regarding reconstruction and/or other processing of theacquired image data can also be provided to the acquisitionconfiguration device 2020 and/or the training device 2010 to improveoperation of the DDLD 2050 for parameter generation. For example, anindication of image quality (e.g., too low, sufficient, too high, etc.)can be processed by the training device 2010 to generate a new deeplearning network to be deployed in the device 2050 to improvedetermination of imaging device 1410 settings by the configurationdevice 2020. At block 840, feedback is collected and stored forprocessing as described above. Feedback is used by the training device2010 to continue to learn, modify, and improve setting recommendations,for example, during operation of the devices 1410, 2020. The feedbackcan be used to train further to respond to good quality and to makesuggestions for bad quality image acquisition, for example.

Thus, the image acquisition configuration device 2020 serves as a blackbox that a customer and/or supplier can provide in conjunction with theimaging device 1410 to configure and improve operation of the imagingdevice 1410 with minimal user interaction. The customer can use andtweak the device 2020 but does not have to train and test the device2020 before deploying the device 2020 with the imaging device 1410. Anaverage user can obtain high quality (or sufficient diagnostic qualityfor reading, etc.) image acquisitions while avoiding a bad quality orunacceptable set of image acquisitions from the imaging device 1410using the configuration device 2020 and its deployed learning device2050.

In certain examples, a single button push can be facilitated for imageacquisition, with the user 1404 pushing the button or otherwiseselecting the option and the configuration device 2020 performing theremaining operations to configure and activate the imaging device 1410.For example, the device 2020 generates DICOM header information to beprovided with patient information to the imaging device 1410 for imageacquisition. The resulting image can be associated with the DICOM headerincluding data such as patient history, blood pressure, etc. Informationregarding prior images, prior diagnosis, information from othermodality(-ies), etc., can be included.

The deployed learning device 2050 is trained to respond to good (orsufficient) quality images and provide suggestions to the user 1404 whena poor (or insufficient) quality image is obtained. The device 2050recognizes good image quality and suggests settings used to obtain agood quality image in a particular circumstance as default settings forthat particular circumstance. When a bad quality image is obtained(e.g., through bad settings, user error, etc.), the device 2050 cansuggest how to recover from the mistake, such as by suggesting differentsettings that can be changed to correct the mistake. Input parametersinclude default field of view (DFOV), center, voltage (kV), current(mA), pitch, orientation, injection rate, injection timing, etc. Ratherthan acquiring a scout image to identify landmarks and use thosesettings, deep learning through the deployed device 2050 can facilitatea one-click determination of scan range, field of view, and/or othersettings, and the operator can modify or approve and activate the imageacquisition, for example.

For example, for a liver study, there are different ways to scan thepatient. By providing symptoms, reasons for ordering the study, bloodtest results, etc., rather than pure operator selection, blood test,etc., the deployed learning device 2050 can determine how to scan thepatient 1406, use or do not use contrast injection (e.g., how fast,concentration, total injection volume, etc.), use or do not use dualenergy, etc. Settings can be evaluated and configured for a plurality ofimaging modalities, such as CT, MICT, SPECT, PET, etc., which follow thesame process with different inputs for different outputs. Cardiacimaging, neuro profusion, lung cancer screening, treatment response,etc., can be supported and improved using the configuration device 2020.

For example, if a patient 1406 is scheduled for a contrast-enhancedliver exam, a patient having normal heart size, size, and liver functionuses certain contrast settings, but if the patient's cardiac function islow (e.g., compromised), a slower bolus (e.g., slower injection rate)and longer diluted contrast can be configured with the imaging device1410 to help ensure that the contrast is provided in a particular placeof the patient. Without more accurate configuration settings provided bythe configuration device 2020, over nine hundred views of the patientmay be obtained. By leveraging prior learning, however, only sixty viewsmay be involved for a diagnosis, for example. In certain examples,working with the system health module 1550 and system design engine1560, if the system is not operating with optimal performance,acquisition and/or reconstruction can be adjusted to obtain diagnosticimage quality even though the imaging device 1410 is not operatingnormally. In certain examples, the training learning device 2010 can beperturbed (e.g., periodically and/or based on an event trigger such as aresult, type of data, type of exam, etc.) to force the device 2010 toreevaluate its nodes, connections, weights, etc.

In certain examples, feedback to and/or from the device 2020 is capturedin storage (e.g., stored and/or buffered in a cloud-based storage,etc.)including input data, actual output, and desired output. When asufficient amount of feedback is received, the training learning network2010 of the device 1120 is retrained in an incremental fashion or newlytrained using the additional feedback data (e.g., based on originalfeedback data plus additional feedback data, etc.) depending on theamount of feedback data received. Thus, the training network 2010 canlearn about user and/or site preferences for the imaging device 1410,etc., over time and react to those preferences for settings, alerts,recommended protocols, etc., by redeploying the deployed learning device2050.

Image Reconstruction Examples

As described above, acquired image data can be reconstructed and/orotherwise processed for machine processing and/or human viewing.However, unless the image data is of sufficient quality for the intendedmachine processing and/or human reading, the image acquisition by theimaging device 1410 is not successful and beneficial to the patient1406.

Image quality is an important parameter for medical imaging. Previously,traditional imaging measurement metrics, such as spatial resolution,temporal resolution, and low-contrast detectability, have been usedextensively by the medical imaging community to compare the performanceof different imaging devices, such as x-ray CT. Recently, there aresignificant efforts in redefining the image quality metrics that can belinked closer to the performance of the task-based results. Theseefforts, however, have met with limited success because of the numerousfactors that impact the image quality such as complex anatomy,object-dependent spatial resolution, dose-dependent spatial resolution,image texture, application dependency, noise and pattern, human visualsystem, image artifacts, anatomy-dependent temporal resolution,object-dependent low contrast detectability (LCD), dose-dependent LCD,etc.

In certain examples, iterative reconstruction makes many measurementmetrics nonlinear and less predictable. For example, a modulationtransfer function (MTF) of iterative reconstructed images is bothobject-contrast-dependent as well as dose-dependent. Therefore, it is nolonger sufficient to quote a single set of MTF numbers for an entire CTsystem, for example. One has to indicate the testing conditions underwhich the MTF has been obtained. This transforms a numerical value intoa complex multi-dimensional variable.

This issue is further complicated by the human visual system. Judgmentof the “quality” of an image can vary from one observer to another. Forexample, each radiologist has his or her preference in the appearance ofthe image based on past experiences. Some radiologists prefer coarsernoise texture while other radiologists prefer finer texture. Often,radiologists link the presence of noise in the image with the“sharpness” of the structure in the image. Additionally, image texturecannot currently be mathematically defined. Many attempts, such as theintroduction of noise-power-spectrum (NPS), fail to differentiate subtledifferences in noise texture, for example.

Given the complexity of the problem, certain examples provide systemsand methods to establish image quality metrics based on deep learningand/or other machine learning. For purposes of illustration only, themethodology is focused on x-ray CT imaging analysis and quality metrics.It should be understood, however, that such technology can be broadlyapplicable to other imaging modalities, such as MR, PET, SPECT, x-ray,ultrasound, etc.

For x-ray CT, an image quality index (IQI) includes a plurality offactors, such as dose, and can be influenced by environmental factorssuch as the level of x-ray flux. In general, a higher x-ray doseproduces better image quality. However, there is a detrimental effect onpatient health since CT uses ionization radiation, and a high level ofradiation exposure is linked to an increased probability of cancer.Therefore, it is desirable to establish IQI as a function of the doseprovided to a patient, such as illustrated in the graph of FIG. 22. Notethat IQI versus dose can be clinical application-dependent. For example,40-50 Milligray (mGy) radiation is used to produce good non-contrasthead images, while 8-10 mGy is used to generate good abdomen and pelvisimages.

In certain examples, IQI is based on a 5-point scale for humanconsumption. In other examples, image quality is generated for computeranalysis as a change in probabilistic values of image classification. Ona scale of 1-5, for example, a 3 indicates the image is diagnosable(e.g., is of diagnostic quality), a 5 indicates a perfect image (e.g.,probably at too high of a dose), and a 1 indicates the image data is notusable for diagnosis. As a result, a preferred score is 3-4. The DDLD1532 can generate an IQI based on acquired image data by mimickingradiologist behavior and the 1-5 scale. Using image data attributes, theDDLD 1532 can analyze an image and determine features (e.g., a smalllesion) and evaluate diagnostic quality of each feature in the imagedata. If the IQI is low (e.g., 1, 2, etc.), the DDLD 1532 can providesuggestions as to how to improve the image quality at the acquisitionDDLD 1522. If the IQI is satisfactory (e. 3, 4, etc.), the image can berecommended for user 1404 (e.g., radiologist, etc.) reading. In certainexamples, the learning and improvement factory 530 can learn about aspecific user and/or site image quality preferences over time. Forexample, Dr. S usually likes to see images with an IQI of 4. Learningthis, the learning and improvement factory 1530 and/or the image qualityassessment engine 1572 can propose a scan protocol to achieve an IQI of4 or trigger a warning that the protocol will not achieve Dr. S's IQIpreference. Thus, the reconstruction learning and improvement factory1530 and/or the image quality assessment engine 1572 can facilitate aself-learning protocol based on the IQI determination (e.g., the factory1530 learns that a user prefers protocol X to reach an IQI of Y, etc.).

In certain examples, the reconstruction DDLD 1532 can model an image ashaving varying probabilities of belong to a certain value or class. Forexample, an image can be categorized as belonging to class 4 with anassociated probability of 90%, a 9% probability that the image belongsto class 5, and a 1% probability that the image belongs to class 3. Thevalue of these percentages over time can be leveraged to statisticallydetermine gradual changes at a more granular level.

While traditional methods of generating IQI have not been successful, atleast because they fail to account for non-linear iterativereconstruction and the less-predictable nature of the human visualsystem, certain examples provide IQI generation that accounts fornon-linear iterative reconstruction and human visualization. Asdescribed above, deep learning can be used to train and refine a targetalgorithm based on input data and desired output(s). Certain examplesapply deep learning and/or other machine learning to imagereconstruction and image quality metric, such as IQI, etc.,determination.

Deep learning tries to mimic the human brain by recognizing objectsusing a layered approach. As the deep learning network is navigated froma lower layer to a higher layer, a higher-level set of features isextracted and abstracted. The extraction and abstraction provide ananswer to the question and an identification of key “features” using todetermine the answer. For example, image features used to determine theIQI include local signal to noise ratio, Markov random fields, scale andspace based Gabor wavelet decomposition, Fourier transforms, etc. Thesefeatures can be used to initialize the neural network and supplementautomated feature maps to generate a classifier for image quality, forexample. Images can be two-dimensional (2D), three-dimensional (3D),four-dimensional (4D), or n-dimensional (ND) images of a variety ofmodalities.

Deep learning input includes labeled images and/or unlabeled images, forexample. The labeled images can be classified based on clinicalapplications, human anatomy, and/or other important attribute. Thelabeled images have also undergone image quality evaluation, and an IQIcan be assigned to each labeled image. The labeled images are ratedbased on a confidence level for making clinical decisions using theimage. For example, a level 3 indicates sufficient confidence to make adecision based on the image, while a level 5 indicates the highest levelof confidence in making the decision based on the image. A level 1, onthe other hand, indicates that such image cannot be used for diagnosis.The labeled images are used to train the deep learning image qualityalgorithm initially.

FIGS. 23A-23B illustrate example learning and testing/evaluation phasesfor an image quality deep learning network. As shown in the example ofFIG. 23A, known, labeled images 2310 are applied to a convolutionnetwork 2320. The images 2310 are obtained using multiple users, andtheir image quality indices are known. As discussed above with respectto FIGS. 1-3, the convolution 2320 is applied to the input images 2310to generate a feature map, and pooling 2330 reduces image size toisolate portions 2325 of the images 2310 including features of interestto form a fully connected layer 2340. A classifier 2350 (e.g., a softmaxclassifier, etc.) associates weights with nodes representing features ofinterest. The classifier 2350 provides weighted features that can beused to generate a known image quality index 2360. In certain examples,a central tendency metric such as average image quality indices can beused as the known image quality index 2360 for training purposes. Theevaluation can be performed separately on individual images, forexample.

In the example of FIG. 23A, a number of feature maps are created byconvolving the input 2310 by a number of convolutional kernels 2320.Each convolutional kernel 2320 is randomly initialized and, as thelearning progresses, the random kernels converge to “feature maps.” Thisis followed by the pooling layer 2330. Fully connected layers 2340 areformed by pooling 2330 and additional convolution and pooling layers maybe optionally added. A classifier stage 2350 is the final layer todetermine the output index 2360. In certain examples, training of thenetwork 2300 is done in batches using a stochastic gradient method(SGD).

Unlabeled images are images that have not yet been evaluated to identifyand label features in the image. Unlabeled images can be used to testthe performance of the deep learning algorithm trained in FIG. 23A andrefine the algorithm performance. As illustrated in the example of FIG.23B, the example network 2300 can also be applied to unlabeled images2315. An image quality index 2365 can be generated and can be comparedto the known image quality index 2360 to evaluate the development andreliability of the network 2300. If the network 2300 is tested and foundto be a satisfactory determiner of image quality, the network 2300 canbe deployed as the reconstruction DDLD 1532, for example.

There are several deep learning and other machine learning techniquesthat can be useful for classifying images to be associated with certainIQI. For example Deep Convolutional Networks can be set up in severalways depending upon the availability of labeled data, computational andmemory constraints, performance requirements, etc. In a convolutionallayer of an example deep convolutional network, an initial layerincludes a plurality of feature maps in which node weights areinitialized using parameterized normal random variables. The featuremaps are followed by a first pooling layer, which is followed by asecond convolution layer, which is then followed by a second poolinglayer, and so on. Subsequent pooling and convolution layers are optionaldepending upon configuration, complexity, type of data, targetenvironment, etc. A final layer is a classification layer using, forexample, a softmax classifier, to evaluate options in the network.

In certain examples, weights and biases for the classification layer areset to 0. An output from the softmax layer is a set of positive numberswhich sum up to 1. In other words, the output from the softmax layer canbe thought of as a probability distribution. Using this distribution,the network can be used to select values for desired hyper parameters.FIGS. 24A-24B show example learning, validation, and testing phases foran example deep convolution network.

As shown in the example of FIG. 24A, if a large set of labeled data isnot available in a collection of medical data, an example network 2400can be trained, validated and tested. In a learning phase 2410, labeledimages are input 2411 to an unsupervised learning layer 2413 (e.g., anauto encoder, etc.). The unsupervised learning layer 2413 initializes afeature space for the input 2411. After processing by the unsupervisedlearning layer 2413, the image information then passes to one or moresupervised learning layers of a convolutional network 2415. As describedabove, feature maps can be created and features can be reduced via theconvolutional layers 2415. The supervised learning layers 2415 arehidden layers in the network 2400 and perform backpropagation. Output isthen classified via a classification layer 2417, which analyzes weightsand biases and generates one or more image quality indices 2419.

In a validation phase 2420, hyper parameters are tuned by inputtingunlabeled images 2421 to the unsupervised learning layer 2413 and thento the supervised learning layers of the convolutional network 2415.After classification 2417, one or more image quality indices 2419 aregenerated.

After tuning parameters during the validation phase 2420, a testingphase 2430 processes an input unlabeled image 2431 using learned layersof the convolution network 2435. The classification layer 2417 producesan image quality index 2439.

If a large set of labeled data is available, a network can be trained,validated, and tested as shown in the example of FIG. 24B. In a learningphase 2440, labeled images are input 2441 to one or more learning layersof a convolutional network 2445. As described above, feature maps can becreated and features can be reduced via the convolutional layers 2445.Output is then classified via a classification layer 2447, whichanalyzes weights and biases and generates one or more image qualityindices 2449.

In a validation phase 2450, hyper parameters are tuned by inputtingunlabeled images 2451 to the learning layers of the convolutionalnetwork 2445. After classification 2447, one or more image qualityindices 2459 are generated.

After tuning parameters during the validation phase 2450, a testingphase 2460 processes an input unlabeled image 2461 using learned layersof the convolution network 2465. The classification layer 2447 producesan image quality index 2469.

While the examples of FIGS. 24A-24B have been illustrated with autoencoder and deep convolutional networks, deep residual networks can beused in the examples as well. In a deep residual network, a desiredunderlying mapping is explicitly defined in relation to stacked,non-linear internal layers of the network. Using feedforward neuralnetworks, deep residual networks can include shortcut connections thatskip over one or more internal layers to connect nodes. A deep residualnetwork can be trained end-to-end by stochastic gradient descent (SGD)with backpropagation, such as described above.

Additionally, deep learning networks can be improved through ongoinglearning and evaluation in operation. In certain examples, an analysisof intermediate layers of the neural network combined with preprocessingof the input data can be used to determine redundancies in the data todrive data generation efficiencies. Preprocessing of data can include,but is not limited to, principal component analysis, waveletdecomposition, Fourier decomposition, matched filter decomposition, etc.Each preprocessing can generate a different analysis, and preprocessingtechniques can be combined based on the structure of the deep learningnetwork under one or more known conditions. A meta-analysis can then beperformed across a plurality of individual analyses (e.g., from eachpreprocessing function performed).

In certain examples, feedback from the deep learning system can be usedto optimize or improve input parameter selection, thereby altering thedeep learning network used to process input (e.g., image data, deviceparameter, etc.) to generate output (e.g., image quality, devicesetting, etc.). Rather than scanning over an entire set of inputparameters to create raw data, a variation of active learning can beused to select a starting parameter space that provides best results andthen randomly decrease parameter values to generate raw inputs thatdecrease image quality but still maintain an acceptable range of qualityvalues. Randomly decreasing values can reduce runtime by processinginputs that have little effect on image quality, such as by eliminatingredundant nodes, redundant connections, etc., in the network.

For example, to use multiple input parameters to process each rawdataset to produce a corresponding output dataset while reducingparameter set values that still maintain the processed output dataset, asearch strategy is employed to navigate through the parameter space.First, parameters used in data processing (e.g., reconstructionparameters, etc.) are determined. As shown in the example of FIG. 25A, atrained network 2500 is leveraged to determine an output quality (e.g.,IQI of reconstructed image datasets) for a starting or initial parameterset 2510 of reconstruction parameters and is used as a baseline. Forexample, the starting parameter set 2510 includes reconstructionparameters Param 0, Param 1, . . . , Param N. Starting with known values(including redundant parameters), a reference IQI can be determined forN datasets.

Since the goal is to reduce the number of parameters, parameter valuesare decreased according to a given strategy (e.g., gradient decent,etc.) until a stop criteria is met (e.g., eliminate known trivialselections providing bad results, etc.). A parameter value selector 2504determines the search strategy limiting the search space and updatingresults. Raw datasets 2506 (e.g., Dataset 0, Dataset 1, . . . , DatasetN) are processed for reconstruction 2508 to produce N reconstructedimage datasets 2510. An IQI comparator 2512 processes each image dataset2510 to generate a feedback value to the parameter value selector 2504.The feedback value is based on a disparity between an averageparameter-based IQI is to a current average parameter-based IQI. Thisprocess is repeated for different datasets to map the general behaviorof the parameter pruning process for each parameter in a normalizedspace. The process is repeated until the set providing the minimalparameter values is identified which still provides acceptable imagequality which can be used as a best available solution.

FIG. 25B illustrates an example system 2501 for image quality assessmentand feedback using a deployed network model. As shown in the example ofFIG. 25B, an acquisition parameter updater and restarter 2520 providesan update to the acquisition engine 1430. A reconstruction parameterupdater and restarter 2522 provides an update to the reconstructionengine 1440. An orchestrator 2524 coordinates among the engines 2520,2522, and an image quality assessment engine with deployed model 2526.An image quality learning and update factory 2528 learns from a trainingimage database 2530 to train a deep learning network model to bedeployed with the image quality assessment engine 2526 (e.g., the imagequality assessment engine 1572, etc.). In operation, the image qualityassessment engine with deployed model 2526 provides information to thetraining image dataset 2530 which can be used in ongoing monitoring andimprovement of the factory 2528, for example. The training image dataset2530 can include image data representing different classes of exampleerror conditions, for example. Using the orchestrator 2524, theacquisition engine 1430 and/or the reconstruction engine 1440 can beupdated and restarted by the acquisition parameter updater and restarter2520 and/or the reconstruction parameters updater and restarter 2522,respectively, for example.

FIG. 25C illustrates an example system configuration 2503 that furtherincludes a detection/diagnosis parameter updater and restarter 2532associated with the diagnostic engine 1450. The example system 2503 alsoincludes a detection/diagnosis assessment engine with deployed model2534. The diagnosis assessment engine with deployed model 2534 isgenerated from a detection/diagnosis learning and update factory 2536leveraging data from the training image database 2530. The trainingimage database 2530 includes data representing different classes ofexample detection and diagnosis of conditions, for example.

In the example of FIG. 25C, the orchestrator 2524 coordinates among theengines 2520, 2522, 2532, and the detection/diagnosis assessment enginewith deployed model 2534. The detection/diagnosis learning and updatefactory 2535 learns from the training image database 2530 to train adeep learning network model to be deployed with the detection/diagnosisassessment engine 2534 (e.g., the diagnosis assessment engine 1574,etc.). In operation, the detection/diagnosis assessment engine withdeployed model 2534 provides information to the training image dataset2530 which can be used in ongoing monitoring and improvement of thefactory 2534, for example. The engine 2534 can operate in conjunctionwith an expert 2538, for example. Using the orchestrator 2524, theacquisition engine 1430, the reconstruction engine 1440, and/or thediagnosis engine 1450 can be updated and restarted by the acquisitionparameter updater and restarter 2520, the reconstruction parametersupdater and restarter 2522, and/or the detection/diagnosis parameterupdater and restarter 2532, respectively, for example.

Certain examples utilize deep learning and/or other machine learningtechniques to compute task-based image quality from acquired image dataof a target. Since humans can visually appreciate a level of imagequality (e.g., noise, resolution, general diagnostic quality, etc.) byviewing the images, an artificial intelligence or learning method (e.g.,using an artificial neural network, etc.) can be trained to assess imagequality. Image quality (IQ) has usually been estimated based on phantomscans using wires, line pairs, and uniform regions (e.g., formed fromair, water, other material, etc.). This requires a separate scan of thephysical phantom by a human operator and reading by a technician and/orradiologist, and it is often not practical to perform multiple phantomscans to measure image quality. Moreover, image quality itself maydepend on the object or patient being scanned. Hence, the image qualitywith a test phantom may not be representative of quality obtained whenscanning an actual patient. Finally, traditional IQ metrics such asfull-width at half maximum (FWHM) of the point spread function (PSF),modulation transfer function (MTF) cutoff frequency, maximum visiblefrequency in line pairs, standard deviation of noise, etc., are notreflecting true task-based image quality. Instead, certain examplesprovide it is impactful to estimate IQ directly from acquired clinicalimages. Certain examples assess image quality using a feature-basedmachine learning or deep learning approach, referred to as a learningmodel. In certain examples, task-based image quality (and/or overallimage quality index) can be computed directly from actual patient imagesand/or object images.

Using images (e.g., clinical images) with a known image quality (IQ) ofinterest, a learning model can be trained. Additional training imagescan be generated by manipulating the original images (e.g. by blurringor noise insertion, etc., to obtain training images with different imagequality). Once the learning model is trained, the model can be appliedto new clinical images to estimate an image IQ of interest.

For example, image input features such as mean, standard deviation,kurtosis, skewness, energy, moment, contrast, entropy, etc., taken fromcropped raw image data and edge map information are combined with one ormore label such as spatial resolution level, spatial resolution value,etc., to form a training set for a machine learning system. The machinelearning network forms a training model using the training set andapplies the model to features obtained from a test set of image data. Asa result, the machine learning network outputs an estimated spatialresolution (e.g., level and/or value) based on the training modelinformation.

In certain examples, a regression and/or classification method can beused to generate image quality metrics by labeling the training datawith an absolute value and/or level of the corresponding image IQmetric. That is, metrics can include quantitative measures of imagequality (e.g., noise level, detectability, etc.), descriptive measuresof image quality (e.g., Likert score, etc.), a classification of imagequality (e.g., whether the image is diagnostic or not, has artifacts ornot, etc.), and/or an overall index of image quality (e.g., an IQI).

In a feature-based machine learning approach, an input to model trainingincludes extracted features from the training image data set. Featureselection can be tailored to an image IQ of interest. Features include,but are not limited to, features based on a histogram of intensityvalues (e.g., mean, standard deviation, skewness, kurtosis, energy,energy, contrast, moment, entropy, etc.). These features can becalculated from raw image data and/or can be extracted after applying adifference filter on the image for local enhancement and/or other imageoperation and/or transformation. These features can be calculated froman entire image, a cropped image, and/or from one or more regions ofinterest (ROIs). Global and/or local texture features based on anadjacency matrix (such as Mahotas Haralick, etc.) can also be included.

In a deep learning (e.g., convolutional neural network)-based approach,a set of features need not be defined. The DLN will identify featuresitself based on its analysis of the training data set. In certainexamples, more data is involved for training (if features have not beenidentified) than with a feature-based machine learning approach in whichfeatures have been identified as part of the input.

Thus, in certain examples, input can include a full image, a croppedimage (e.g., cropped to a region of interest), an image patch, etc. Withan image patch, smaller image patches can be used to assess imagequality on a local basis, and a map of image quality can be generatedfor the image. Metrics such as quantitative image IQ, such as spatialresolution, noise level, and/or task-based IQ metric (e.g.,detectability, etc.) can be extracted directly from clinical images.Certain examples can be applied in any context in which image qualityassessment is performed or desired, such as to compare imagingtechnologies (e.g., hardware and/or algorithms); during imageacquisition to improve or optimize a scanning technique andreconstruction in real time (or substantially real time given aprocessing, storage, and/or data transmission latency) while reducing orminimizing radiation dose, and/or to provide quantified image IQ toclinicians to help with diagnosis, etc. The proposed techniques can beapplied to other imaging modalities between the example of CT, such asMRI, PET, SPECT, X-ray, tomosynthesis, ultrasound, etc.

Thus, by identifying sources of variation (e.g., in image resolution,etc.) and reconstructing images in view of the variation, astandardization transform can be created by a machine learning network,refined, and applied to image reconstruction (e.g., using thereconstruction engine 1430 and associated DDLD 1532. When imageattributes are computed, a recalibration transform can be developed,refined, and applied to compute image attributes. Analytics can beprovided to clinicians and used to evaluate image resolution using thelearning network.

For example, suppose a data set includes nine cardiac volumes with 224images per volume. A Gaussian blur is applied to the images to generateimages at four additional resolution levels. The total sample size isthen 224*5*9=10080. Seven features are extracted from the raw imagedata, and eight features are extracted from an edge map of the image.Cross-validation is facilitated by splitting the sample into a trainingset (70%) and a test set (30%), and a random forest regression was used.

Results were generated and contrasted between estimated (using machinelearning) and actual (measured) error or distribution. For example, FIG.26 illustrates a comparison of estimated FWHM (in millimeters) to trueFWHM. FIG. 27 shows an example true FWHM distribution. FIG. 28 shows anexample estimated FWHM distribution. FIG. 29 shows an example estimationerror in the FWHM of the PSF (e.g., estimated FWHM—true FWHM (mm)). FIG.30 shows an example comparison of feature importance from the exampledata set. The example graph of FIG. 30 organizes feature importance byfeature index. Features from raw image data include: 0: Mean, 1:Kurtosis, 2:Skewness, 3:Energy, 4:Moment, 5:Contrast, 6:Entropy.Features from the edge map include: 7: Mean, 8: Standard Deviation, 9:Kurtosis, 10:Skewness, 11:Energy, 12:Moment, 13:Contrast, 14:Entropy. Asshown from the example data, machine learning yields reasonable resultsfor estimating spatial resolution from clinical datasets. Additionally,entropy of the edge map is shown to be an important feature to estimatespatial resolution, as well as entropy of the raw image.

Additional use cases can include lung nodule/calcification or smallstructure detection and analysis for lung cancer detection. Source ofvariation can include noise (e.g., mA, peak kilovoltage (kVp), patientsize, etc.), resolution (e.g., reconstruction kernel type, thickness,pixel size, etc.), respiratory and cardiac motion (e.g., rotation speedand patient compliance, etc.), blooming artifact (e.g., reconstructionmethod, partial volume, motion, etc.). An impact on outcome can includemeasurement error in volume and density which lead to under-staging andmissed structure. Another use case can include a cardiac perfusionanalysis to diagnose coronary artery disease (CAD). Source of variationcan include patient physiology (e.g., cross patients and same patient,dynamic range small, etc.), beam hardening artifact (patient uptake,bolus timing, etc.), cardiac motion, contrast pooling, etc. Impact onoutcome can include an incorrect perfusion map (e.g., missed perfusiondefect or wrong diagnosis of perfusion defect, etc.). Another use casecan include liver lesion/small dark structures for cancer detection.Source of variation can include noise (e.g., mA, kVp, patient size,etc.), resolution (e.g., reconstruction kernel type, thickness, pixelsize, etc.), structured noise (e.g., streaks, pattern, texture, etc.),shadowing artifact (e.g., bone, ribs, spines reconstruction artifact,etc.), motion, etc. An impact on outcome can include a missed lesion orincorrect diagnosis due to low contrast detectability.

Another use case can include coronary/vascular imaging. Source ofvariation can include streak or blooming artifact (e.g., reconstructionmethod, partial volume, motion, etc.), noise (e.g., mA, kVp, patientsize, etc.), resolution, etc. Impact on outcome can include, if analysisof lumen is needed, noise and resolution have a bigger impact.

Another use case can include brain perfusion for stroke. Source ofvariation can include shadowing artifact from bone, small physiologicalchange (e.g., dynamic range small, etc.), structured noise (e.g.,reconstruction method, etc.), etc. Impact on outcome can include anincorrect perfusion map (e.g., missed perfusion defect or wrongdiagnosis of perfusion defect, etc.), etc.

Another use case can include Chronic Obstructive Pulmonary Disease(COPD) and/or other lung disease (e.g., pneumoconiosis, etc.) diagnosisand classification (e.g., Thoracic VCAR), etc. Source of variation caninclude noise (e.g., mA, kVp, patient size, slice thickness, etc.),resolution (e.g., kernel, pixel size, thickness size, etc.), contrast(e.g., iodine, etc.), patient physiology (e.g., lung volume during scan,can be measured from image, etc.), respiratory motion, etc. Impact onoutcome can include measurement error (e.g., airway diameter/perimeter,luminal narrowing underestimated, wall thickening overestimated, etc.),etc.

Another use case can include liver fat quantification (e.g., steatosisgrading, cirrhosis staging, etc.). Source of variation can include noise(e.g., mA, kVp, patient size, etc.), resolution (e.g., reconstructionkernel type, thickness, pixel size, etc.), structured noise (e.g.,streaks, pattern, texture, etc.), shadowing artifacts (e.g., ribs,spines reconstruction artifact, etc.), etc. An impact on outcome caninclude measurement error and mis-staging, etc. Another use case caninclude volume/size quantification of other organs (e.g., kidneytransplant, etc.) or masses in organ (e.g., cyst or stones, etc.), etc.

FIG. 31A illustrates a flow diagram of an example method 3100 for imagereconstruction. At block 3102, image data is received from theacquisition engine 1430. For example, the reconstruction engine 1440receives image from the imaging device 1410 via the acquisition engine1430. At block 3104, the image data is pre-processed. For example, theDDLD 1532 pre-processes the image data according to one or more settingsor parameters, such as whether a human-viewable and/or machine-readableimage is to be generated from the acquired image data. For example, thereconstruction DDLD 1532 can be deployed with a network trained toreplace a noise reduction algorithm (e.g., by training the DLN in thelearning and improvement factory 1530 on a plurality of examples ofnoisy and noise-free image pairs) to convert noisy data to produce highquality data.

As described above, a machine-readable image need not be formatted forhuman viewing but can instead be processed for machine analysis (e.g.,computer-aided diagnosis, etc.). Conversely, a human-viewable imageshould have clarity in features (e.g., sufficient resolution and reducednoise, etc.) such that a radiologist and/or other human user 1404 canread and evaluate the image (e.g., perform a radiology reading). TheDDLD 1532 can evaluate the image data before reconstruction anddetermine reconstruction settings, for example. Reconstruction and/orother processing parameters can be determined by the DDLD 1532 forhuman-viewable and/or machine-readable images.

At block 3106, the reconstruction settings are evaluated to determinewhether a human-viewable and/or machine-readable image is to begenerated. In some examples, only a human-viewable image is to begenerated for user 1404 review. In some examples, onlymachine-processable image data is to be generated for automaticevaluation by the diagnosis engine 1450, for example. In some examples,both human-viewable image and machine-processable image data are to beprovided.

If a human-reviewable image is desired, then, at block 3108, an image isreconstructed using the image data for human viewing (e.g., radiologistreading). For example, rather than employing a computationally intensiveiterative reconstruction algorithm that takes in raw data and producesan image, the reconstruction engine 1440 and DDLD 1532 (e.g., trained ona plurality of examples of raw and reconstructed image pairs) canprocess raw image data and produce one or more reconstructed images ofequivalent or near-equivalent quality to the iterative algorithm.Additionally, as described above, the DDLD 1532 can convert noisy datainto higher quality image data, for example. Further, the DDLD 1532 canbe used to condition the image data and provide a “wide view” toreconstruct images outside the field of view (FOV) of a detector of theimaging device 1410. Rather than using equations to extrapolate dataoutside the detector, the DDLD 1532 can fill in the gaps based on whatit has learned from its training data set. If machine-reviewable imagedata is desired, then, at block 3110, the image data is processed formachine analysis. The DDLD 1532 can process the image data to removenoise, expand field of view, etc., for example,

At block 3112, the reconstructed image is analyzed. For example, theimage is analyzed by the DDLD 1532 for quality, IQI, data quality index,other image quality metric(s), etc. The DDLD 1532 learns from thecontent of the reconstructed image (e.g., identified features,resolution, noise, etc.) and compares to prior reconstructed images(e.g., for the same patient 1406, of the same type, etc.). At block3114, the reconstructed image is sent to the diagnosis engine 1450. Theimage can be displayed and/or further processed by the diagnosis engine1450 and its DDLD 1542 to facilitate diagnosis of the patient 1406, forexample.

Similarly, at block 3116, the processed image data is analyzed. Forexample, the image data is analyzed by the DDLD 1532 for quality, IQI,data quality index, other image quality metric(s), etc. The DDLD 1532learns from the content of the machine-processable image data (e.g.,identified features, resolution, noise, etc.) and compares to priorimage data and/or reconstructed images (e.g., for the same patient 1406,of the same type, etc.). At block 3118, the processed image data is sentto the diagnosis engine 1450. The image data can be further processed bythe diagnosis engine 1450 and its DDLD 1542 to facilitate diagnosis ofthe patient 1406, for example. For example, machine-readable image datacan be provided to the diagnosis engine 1450 along with other patientinformation (e.g., history, lab results, 2D/3D scout images, etc.) whichcan be processed together to generate an output to support the user 1404in diagnosing the patient 1406 (e.g., generating support documentationto assist the radiologist in reading the images, etc.).

FIG. 31B provides further detail regarding blocks 3112 and 3116 in aparticular implementation of the example method 3100 of FIG. 31A forimage reconstruction. The example method of FIG. 31B can be triggered byone or both of blocks 3112 and 3116 in the example method 3100.

At block 3120, the image/image data is analyzed to determine whether theacquired image is a good quality image. To determine whether theacquired image data represents a “good quality” image, the data can becompared to one or more thresholds, values, settings, etc. As describedabove, an IQI, other data quality index, detectability index, diagnosticindex, etc., can be generated to represent a reliability and/orusefulness of the data for diagnosis of the patient 1406. While the IQIcaptures a scale (e.g., a Likert scale, etc.) of acceptability of animage to a radiologist for diagnosis. Other indices, such as resolutionimage quality, noise image quality, biopsy data quality, and/or otherdata quality metric can be incorporated to represent suitability ofimage data for diagnosis, for example. For example, a task-specific dataquality index can represent a quality of acquired image data formachine-oriented analysis.

At block 3122, if the acquired and processed image and/or image data isnot of sufficient quality, then the reconstruction DDLD 1532 sendsfeedback to the acquisition learning and improvement factory 1520indicating that the image data obtained is not of sufficient quality foranalysis and diagnosis. That way the factory 1520 continues to learn andimprove image acquisition settings for different circumstances and cangenerate a network model to redeploy the DDLD 1522. At block 3124, theacquisition engine 1430 triggers a re-acquisition of image data from thepatient 1406 via the imaging device 1410 (e.g., at block 3102). Thus,the reconstruction DDLD 1532 and acquisition DDLD 1522 can work togetherto modify imaging parameters and reacquire image data while the patient1406 may still be on the table or at least in close proximity to theimaging device 1410, for example, thereby reducing hardship on thepatient 1406 and staff as well as equipment scheduling.

At block 3126, if the image/image data quality satisfies the threshold,the image quality can be evaluated to determine whether the quality istoo high. An image quality that is too high (e.g., an IQI of 5indicating a “perfect” image, etc.) can indicate that the patient 1406was exposed to too much radiation when obtaining the image data. If animage quality of 3 or 4 is sufficient for diagnostic reading by the user1404 and/or diagnosis engine 1450, for example, then an image quality of5 is not necessary. If the image quality is too high, then, at block3128, feedback is provided from the reconstruction DDLD 1532 to theacquisition learning and improvement factory 1520 to adjustdosage/intensity settings of the imaging device 1410 for future imageacquisition (e.g., of a particular type, for that patient, etc.). Theprocess then continues at block 3114 and/or 3118 to provide thereconstructed image (block 3114) and/or processed image data (block3118) to the diagnosis engine 1450 for processing and review.

System Health and System Improvement

As described above, the system design engine 1560 builds and maintainsone or more digital models of the system 1400, 1500 and/or itsindividual components 1410, 1420, 1430, 1440, 1450, etc. The systemdesign engine 1560 also evaluates an indication of system health fromthe system health module 1550 to identify potential issues, problems,areas for improvement, etc. For example, an indication of poor systemhealth by the system health module 1550 based on its processing offeedback from the engines 1430, 1440, 1450 can trigger an analysis ofgeneral design improvement, for example.

As shown in the example of FIG. 32A, the system design engine 1560includes an input formatter 1561, a model processor 1563, one or moretraining deep learning network models 1565, and an output generator1569. In certain examples, a plurality of target system 1500 componentsare connected to the engine 1560, and, a DLN can be associated with eachcomponent. Each DLN involves known inputs and outputs to train it. Theeinputs and outputs simulate inputs and outputs of the physicalcomponent. DLNs can be connected like components to derive a digitalmodel of the target system 1500. Using the digital model,recommendation(s) can be provided based on simulations run on thenumerical model by the model processor 1563.

The example system design engine 1560 includes a system digital modellibrary 1580 including a plurality of component models (e.g., anacquisition digital model 1581, reconstruction digital model 1583,diagnosis digital model 1585, etc.) and a composite system digital model1587. The models 1581-1587 are generated and deployed using the trainingdeep learning network models 1565. The models 1581-1585 can be connectedas their corresponding system 1400, 1500 components are connected toform a digital model of the target system (e.g., the composite model1587, etc.). As illustrated in the example of FIG. 32B, each model (theexample shown here being the composite system model 1587) is implementedusing an input 3202 to receive input data, parameter, instruction, etc.,a deployed deep learning network model 3204 generated using a traineddeep learning network to process the input and produce an output, whichis taken by an output 3206 and provided to the model processor 1563and/or output generator 1565 to be used in a recommendation for one ormore components of the target system 1400, 1500 being monitored.

The system design engine 1560 leverages system inputs and outputs. Thesystem design engine 1560 includes DLNs 1581-1587 that are trained,validated and tested using sufficiently large datasets of knowncomponent inputs and outputs that can adequately represent thevariability in the expected data the system design engine 1560encounters through operation of the system 1500 and its components. TheDLN models 1581-1587 receive input from the overall system 1500 and/orits individual components 1410-1450, 1520-1540, 1522-1542, 1550, 1552,1555, etc. The input formatter 1561 processes the input to normalizeand/or properly format the input, validate/verify the input, supplementthe input, etc. The models in the model library 1580 work with the modelprocessor 1563 to process the input and simulate operation of the system1500 and/or its components 1410-1450 using the models 1580. The trainingmodels 1565 can continue to receive feedback to modify the model(s) 1565for redeployment of the models 1581-1587 in the library 1580, forexample.

In certain examples, system health 1550 input 1561 helps the DLN models1580 to model operation and status of the system 1500 to developmaintenance and/or replacement schedules based on usage schedules,patterns, device 1410, 1420, 1430, 1440, 1450 status, etc. For example,feedback regarding declining image quality from the reconstruction DLN1532 can be reflected by the system health module 1550 and provided asinput 1561 to update the training model(s) 1565 and affect theacquisition digital model 1581 to generate and/or modify a maintenanceschedule, replacement timeline, etc. (e.g., because an x-ray tube isfailing, a detector is going bad, etc.).

In other examples, feedback from the DDLDs 1522, 1532, 1542 can suggestor be used to identify a design limitation in existing equipment, suchas the imaging device 1410, information subsystem 1420, etc. Forexample, consistently off-target image acquisition may indicate to theDDLD 1552 that a patient positioner in the imaging device 1410 cannotproperly position patients of a certain size. As the DLN device 1552 andassociated system learning and improvement factory 1555 gather feedbackof this type over time and makes connections between image quality,patient positioning, and patient size, one or more of the models1581-1587 in the library 1580 can determine the relationship betweenthese factors and suggest a variation in the physical design of theimaging device 1410, for example.

In certain examples, a recommendation from the engine 1560 and/or systemhealth module 1550 can be used to adjust an imaging and/or otherexamination protocol. For example, a “standard” or default imagingprotocol can be provided for the imaging device 1410 at a certain site.However, the particular site and its equipment and operators may havecertain preferences, constraints, etc. The system design engine 1560,for example, processes the information it receives and learnsconsistencies and inconsistencies with the default protocol. The engine1560 can then suggest and/or automatically make changes to the imagingprotocol for the site (e.g., with an option for user overridden and/oradjustment).

For example, an exam protocol can be adjusted based on the state of thepatient 1406 at the time of the exam. Patient heart rate and heart ratevariability can be used to adjust contrast volume and/or timing, forexample. Patient dimensions can determine optimal kV, mA, and pitchsettings. These settings can also be adjusted for institutionalpreferences (e.g., adjust noise levels, mA settings, etc.) in acontinual learning feedback loop using the design engine 1560.Recommendation can then be provided to modify individual componentsusing associated DLNs 1581, 1583, 1585.

In certain examples, machine health monitoring can be facilitated usingthe system design engine 1560. For example, a data quality index such asan IQI, etc., can be used by the engine 1560 to standardize or normalizedata, and the design engine 1560 can monitor probabilities of acomponent belong to one or more classes or categories (e.g., monitoringthat the imaging device 1410 is most likely providing images ofacceptable quality, more likely providing images of unacceptable qualityfor diagnosis, etc.). The design engine 1560 can also analyze data logfiles, audio recordings of user-patient interaction, audio recordings ofmachine noises, customer feedback datasets, etc., to monitor andevaluate machine health (e.g., health of the imaging device 1410, etc.).In certain examples, the system design engine 1560 can computenormalized deviations (e.g., z-scores) of current machine values fromcorresponding “normal” or accepted values, etc.

In certain examples, system 1400, 1500 design improvements can begenerated by the system design engine 1560 based on an analysis ofmaintenance and/or service issues from machines, such as the imagingdevice 1410, deployed in the field. Data can be retrieved from theimaging device 1410, for example, via the learning and improvementfactories 1520, 1530, 1540 and/or system health module 1550 and providedto the model library 1580 via the input formatter 1561. The modelprocessor 1563 works with one or more models 1581-1587 from the library1580 to process the information (e.g., simulate operation and possiblevariation(s) in outcome, parameter setting, configuration, etc.) tosuggest future design improvements for the imaging device 1410. Data canbe processed for one or more imaging devices 1410 based on model number,modality, customer use type, etc. Additional text sources such aspapers, patents, Web content, etc., can also be added to one or moremodels in the model library 1580 via the training deep learning networkmodel(s) 1565, used to re-train and re-deploy one or more models1581-1587 in the library. Imaging devices 1410 (also referred to asscanners) can be differentiated based on their respective capabilitiesand usage statistics. The model(s) 1581-1587 can identify patterns andrelationships and help to quantify why a certain scanner should bebought and/or used. This can be quantified using a scanner qualityindex, scanner value index, etc. (e.g., a rating of 1-5 with 5 beingmost useful for a particular system/application and 1 being least usefulfor a particular system/application, etc.). Thus, the system designengine 1560 can facilitate competitive benchmarking. Based on use, thesystem design engine 1560 can determine what is needed to improve futuresystem 1500 design(s), including whether or not a new scanner should bebought and which scanner should be bought, etc.

In certain examples, machine repair scheduling can be supported andenhanced using the system design engine 1560. Information can beretrieved from the imaging device 1410 and/or its learning andimprovement factories 1520, 1530, 1540 to identify problems, errors,faults, inefficiencies, insufficiencies, overages, etc., and the systemdesign engine 1560 can help the factories 1520, 1530 and/or 1540, andassociated DDLDs 1522, 1532, 1542, adjust imaging device 1410 parametersand/or otherwise compensate for issues with the imaging device 1410and/or other system 1500 component based on its processing ofinformation and system health 1550, for example. Thus, the system 1500can be self-healing for many issues. If an issue involves hardwaremaintenance and/or replacement, the system design engine 1560 can helppredict and schedule the maintenance and/or replacement throughautomatic scheduling, notification of the user 1404, error logging, etc.

Thus, a deep learning network model 1581-1587 is associated with eachcomponent of a target system 1400, 1500, to be emulated, and each deeplearning network model is trained using known input and known outputwhich simulate input and output of the associated component of thetarget system. Each deep learning network model is connected as eachassociated component to be emulated is connected in the target system toform a digital model of the target system. The model processor 1563simulates behavior of the target system and/or each component of thetarget system to be emulated using the digital model to generate arecommendation regarding a configuration of a component of the targetsystem and/or a structure of the component of the target system.

FIG. 33 illustrates a flow diagram of a method 3300 to monitor andimprove system health, configuration, and/or design. At block 3302, dataregarding component operation and system health is gathered via theinput formatter 1561. For example, system health information can begathered via the system health module 1550, factories 1520, 1530, 1540,devices 1410, 1420, engines 1430, 1440, 1450, etc.

At block 3304, one or more training learning network models 1565 aretrained based on input. For example, known input corresponding to knownoutput for each component of the target system being monitored is usedto train behavior of the corresponding model 1565 until the model 1565is stable and predictable for deployment in the library 1580. In certainexamples, input gathered via the system health module 1550, factories1520, 1530, 1540, devices 1410, 1420, engines 1430, 1440, 1450, etc.,can be used to continue to train the model(s) 1565 on the particularsystem and its components. At block 3306, on the model(s) 1565 have beentrained, the model(s) 1565 are used to generate deployed models1581-1587 in the model library 1580.

At block 3308, system operation is monitored and modeled using thesystem design engine 1560. For example, gathered input data is formattedby the input formatter 1561 and processed using the model processor 1563in conjunction with one or more models 1581-1587 from the model library1580 associated with each component being modeled and monitored. Forexample, the data is used to form and/or modify nodes and/or connectionsbetween nodes in a deep learning network, such as a deep convolutionalneural network, auto-encoder network, deep residual network, machinelearning network, etc., embodied in one or more models 1581-1587 in thelibrary or catalog 1580 (as described above). Weights and/or biasesassociated with nodes, connections, etc., can also be modified bypatterns, relationships, values, presence or absence of values, etc.,found by the model(s) 1581-1587 in the input data, for example. Eachmodel 1581-1587, taken alone or in combination (e.g., connected as thecorresponding system components are connected in the target system 1500to form a digital model, digital twin, etc.), can be used by the modelprocessor 1563 to simulate component(s) and/or overall system 1500operation, given the received input.

Thus, the system design engine 1560 can use input data and the model(s)in the library 1580 to simulate operation of the imaging device 1410and/or other component and predict result, failure, maintenanceschedule, etc. As more data is gathered from actual operation of theimaging device 1410, the training network model(s) 1565 can be updatedfor improved modeling and understanding of the device 1410 to generatemore accurate deployed network models in the model library 1580, forexample.

At block 3310, a recommendation is generated by the model processor 1563based on simulation using the model(s) 1581-1587. For example,maintenance/repair for the imaging device 1410, a change inconfiguration setting for the imaging device 1410, a suggested physicalmodification and/or new product feature, etc., can be recommended basedon processing of system operation information by the system designengine 1560.

At block 3312, the recommendation is analyzed to determine whether thechange impacts component configuration and/or component structure. Ifthe observed change impacts component configuration, then, at block3314, the system design engine 1560 can process the change andsuggest/generate a correction to the configuration, for example. If theobserved change impacts component structure, then, at block 3316, aproposed design change can be generated for subsequent development(e.g., by a design team).

If the recommendation is a configuration change, then, at block 3314, anoutput recommendation 1569 is generated regarding configuration of oneor more system component(s). For example, a request formaintenance/repair for the imaging device 1410, a change inconfiguration setting for the imaging device 1410, etc., can be output1569 to the imaging device 1410, DDLD 1522, learning and improvementfactory 1520, information subsystem 1420, and/or external system forimplementation, publication, further processing, etc. In certainexamples, based on receipt of the output recommendation, one or more ofthe learning and improvement factories 1520, 1530, 1540 is modified. Forexample, the factories 1520, 1530, 1540 learn from the processing andrecommendation generated by the system design engine 1560 to improvetheir information, understanding, and operation. If the outputrecommendation from the system design engine 1560 includes a change inparameter and/or other setting, the corresponding factory 1520, 1530,1540 modifies node(s), weight(s), connection(s), bias(es), etc., of itsincluded DLN based on that recommendation. Thus, the factories 1520,1530, 1540, 1555 continue to learn from each other in a feedback loopand continue to evolve and provide better output for their correspondingcomponents.

If the recommendation is a structural change, then, at block 3316, anoutput 1569 recommendation is generated with a suggestion regardingfuture design changes to and/or physical arrangement of one or moresystem components. For example, maintenance/repair for the imagingdevice 1410, a suggested physical modification and/or new productfeature, etc., can be output 1569 to the imaging device 1410, DDLD 1522,learning and improvement factory 1520, information subsystem 1420,and/or external system for implementation, publication, furtherprocessing, etc. For example, based on the output 1569 recommendationand/or further feedback from the corresponding digital factory 1520,1530, 1540, one or more of the imaging device 1410, informationsubsystem 1420, acquisition engine 1430, reconstruction engine 1440,diagnostic engine 1450, etc., can be modified (e.g., physicalconfiguration and/or design changed, etc.) and/or reconfigured (e.g., asetting or parameter changed, etc.).

At block 3318, the input and output are evaluated to determine whether atraining model 1565 should be adjusted. For example, a deviation notedin the input can be used to update the model(s) 1565, potentiallyresulting in redeployment of one or more models 1581-1587 in thedeployed model library 1580, for example. If not, monitoring of systemoperation continues.

Thus, machine health impacts patient health, and the system designengine 1560 can monitor, model, and evaluate machine health and triggerchanges to machine configuration to improve machine operation and,thereby, help avoid potential negative impact on patient health. Forexample, the system design engine 1560 can instruct the acquisitionlearning and improvement factory 1520 and/or the acquisition engine 1430itself (and its DDLD 1522) to adjust settings, slow down pitch, adjustcontrast, etc., to help ensure desired images are obtained at a desiredimage quality. The system design engine 1560 understands machine statusand capabilities and can help the learning and improvement factory 1520learn to react accordingly, for example. Machine learning through theengine 1560 at the system level can also be used to leverage learning ofsystem and patient patterns to drive patient behavior throughadjustments to protocol, workflow, order of operations, device settings,etc. Each can be mapped as a node in the network, and different nodescan be weighted differently based on device characteristic, desiredoutcome, relationship, etc. In certain examples, medical devices and/orother devices outside of and/or ancillary to the system 1500 can bemodeled and modified, such as pacemakers, baby warmers, fitnesstrackers, biometric sensors, etc.

FIG. 34 illustrates an example representation of data flow 3400 betweenthe system design engine 1560 and other system 1500 components such asthe imaging device 1410, acquisition engine 1430, system health module1550, and an external system 3420. The acquisition engine 1430 and itsDDLD 1522 interact with the device 1410 to configure 3402 the device1410 and obtain feedback 3404 from the device 1410. The system healthmodule 1550 monitors feedback 3406 from the acquisition engine 1430, andprovides feedback 3410 to the system design engine 1560. The systemdesign engine 1560 provides a recommendation 3412 to the system healthmodule 1550 regarding configuration of the device 1410, which is routed3408 by the system health module 1550 to the acquisition engine 1430,which, in turn, provides 3402 information to the device 1410. The systemdesign engine 1560 can also provide a recommendation 3414 to an externalsystem 3420 for a change in physical design and/or configuration for thedevice 1410. Thus, the components 1410, 1430, 1550, 1560 are in afeedback loop for ongoing monitoring, processing, and improvement.

In certain examples, the acquisition engine 1430 (and its DDLD 1522) canlearn about a specific user and/or site image quality preferences overtime. The engine 1430 can propose a scan protocol to achieve the learnedpreference or trigger a warning when the preference will not beachieved. Thus, the engine 1430 (in conjunction with its DDLD 1522 andfactory 1530) can facilitate a self-learning protocol based on the IQIdetermination (e.g., learning that a user/site prefers protocol X toreach an IQI of Y, etc.).

While example implementations are illustrated in conjunction with FIGS.1-34, elements, processes and/or devices illustrated in conjunction withFIGS. 1-34 may be combined, divided, re-arranged, omitted, eliminatedand/or implemented in any other way. Further, components disclosed anddescribed herein can be implemented by hardware, machine readableinstructions, software, firmware and/or any combination of hardware,machine readable instructions, software and/or firmware. Thus, forexample, components disclosed and described herein can be implemented byanalog and/or digital circuit(s), logic circuit(s), programmableprocessor(s), application specific integrated circuit(s) (ASIC(s)),programmable logic device(s) (PLD(s)) and/or field programmable logicdevice(s) (FPLD(s)). When reading any of the apparatus or system claimsof this patent to cover a purely software and/or firmwareimplementation, at least one of the components is/are hereby expresslydefined to include a tangible computer readable storage device orstorage disk such as a memory, a digital versatile disk (DVD), a compactdisk (CD), a Blu-ray disk, etc. storing the software and/or firmware.

Flowcharts representative of example machine readable instructions forimplementing components disclosed and described herein are shown inconjunction with at least FIGS. 8C, 8D, 12, 13, 16, 17, 19, 21, 31A,31B, and 33. In the examples, the machine readable instructions includea program for execution by a processor such as the processor 3512 shownin the example processor platform 3500 discussed below in connectionwith FIG. 35. The program may be embodied in machine readableinstructions stored on a tangible computer readable storage medium suchas a CD-ROM, a floppy disk, a hard drive, a digital versatile disk(DVD), a Blu-ray disk, or a memory associated with the processor 3512,but the entire program and/or parts thereof could alternatively beexecuted by a device other than the processor 3512 and/or embodied infirmware or dedicated hardware. Further, although the example program isdescribed with reference to the flowcharts illustrated in conjunctionwith at least FIGS. 8C, 8D, 12, 13, 16, 17, 19, 21, 31A, 31B, and 33,many other methods of implementing the components disclosed anddescribed herein may alternatively be used. For example, the order ofexecution of the blocks may be changed, and/or some of the blocksdescribed may be changed, eliminated, or combined. Although theflowcharts of at least FIGS. 8C, 8D, 12, 13, 16, 17, 19, 21, 31A, 31B,and 33 depict example operations in an illustrated order, theseoperations are not exhaustive and are not limited to the illustratedorder. In addition, various changes and modifications may be made by oneskilled in the art within the spirit and scope of the disclosure. Forexample, blocks illustrated in the flowchart may be performed in analternative order or may be performed in parallel.

As mentioned above, the example processes of at least FIGS. 8C, 8D, 12,13, 16, 17, 19, 21, 31A, 31B, and 33 may be implemented using codedinstructions (e.g., computer and/or machine readable instructions)stored on a tangible computer readable storage medium such as a harddisk drive, a flash memory, a read-only memory (ROM), a compact disk(CD), a digital versatile disk (DVD), a cache, a random-access memory(RAM) and/or any other storage device or storage disk in whichinformation is stored for any duration (e.g., for extended time periods,permanently, for brief instances, for temporarily buffering, and/or forcaching of the information). As used herein, the term tangible computerreadable storage medium is expressly defined to include any type ofcomputer readable storage device and/or storage disk and to excludepropagating signals and to exclude transmission media. As used herein,“tangible computer readable storage medium” and “tangible machinereadable storage medium” are used interchangeably. Additionally oralternatively, the example processes of at least FIGS. 8C, 8D, 12, 13,16, 17, 19, 21, 31A, 31B, and 33 may be implemented using codedinstructions (e.g., computer and/or machine readable instructions)stored on a non-transitory computer and/or machine readable medium suchas a hard disk drive, a flash memory, a read-only memory, a compactdisk, a digital versatile disk, a cache, a random-access memory and/orany other storage device or storage disk in which information is storedfor any duration (e.g., for extended time periods, permanently, forbrief instances, for temporarily buffering, and/or for caching of theinformation). As used herein, the term non-transitory computer readablemedium is expressly defined to include any type of computer readablestorage device and/or storage disk and to exclude propagating signalsand to exclude transmission media. As used herein, when the phrase “atleast” is used as the transition term in a preamble of a claim, it isopen-ended in the same manner as the term “comprising” is open ended. Inaddition, the term “including” is open-ended in the same manner as theterm “comprising” is open-ended.

FIG. 35 is a block diagram of an example processor platform 3500structured to executing the instructions of at least FIGS. 8C, 8D, 12,13, 16, 17, 19, 21, 31A, 31B, and 33 to implement the example componentsdisclosed and described herein. The processor platform 3500 can be, forexample, a server, a personal computer, a mobile device (e.g., a cellphone, a smart phone, a tablet such as an iPad™), a personal digitalassistant (PDA), an Internet appliance, or any other type of computingdevice.

The processor platform 3500 of the illustrated example includes aprocessor 3512. The processor 3512 of the illustrated example ishardware. For example, the processor 3512 can be implemented byintegrated circuits, logic circuits, microprocessors or controllers fromany desired family or manufacturer.

The processor 3512 of the illustrated example includes a local memory3513 (e.g., a cache). The example processor 3512 of FIG. 35 executes theinstructions of at least FIGS. 8C, 8D, 12, 13, 16, 17, 19, 21, 31A, 31B,and 33 to implement the learning and improvement factories 1520, 1530,1540, 1555 and/or other components such as information subsystem 1420,acquisition engine 1430, reconstruction engine 1440, diagnosis engine1450, etc. The processor 3512 of the illustrated example is incommunication with a main memory including a volatile memory 3514 and anon-volatile memory 3516 via a bus 3518. The volatile memory 3514 may beimplemented by Synchronous Dynamic Random Access Memory (SDRAM), DynamicRandom Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM)and/or any other type of random access memory device. The non-volatilememory 3516 may be implemented by flash memory and/or any other desiredtype of memory device. Access to the main memory 3514, 3516 iscontrolled by a clock controller.

The processor platform 3500 of the illustrated example also includes aninterface circuit 3520. The interface circuit 3520 may be implemented byany type of interface standard, such as an Ethernet interface, auniversal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices 3522 are connectedto the interface circuit 3520. The input device(s) 3522 permit(s) a userto enter data and commands into the processor 3512. The input device(s)can be implemented by, for example, a sensor, a microphone, a camera(still or video), a keyboard, a button, a mouse, a touchscreen, atrack-pad, a trackball, isopoint and/or a voice recognition system.

One or more output devices 3524 are also connected to the interfacecircuit 3520 of the illustrated example. The output devices 3524 can beimplemented, for example, by display devices (e.g., a light emittingdiode (LED), an organic light emitting diode (OLED), a liquid crystaldisplay, a cathode ray tube display (CRT), a touchscreen, a tactileoutput device, and/or speakers). The interface circuit 3520 of theillustrated example, thus, typically includes a graphics driver card, agraphics driver chip or a graphics driver processor.

The interface circuit 3520 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network3526 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 3500 of the illustrated example also includes oneor more mass storage devices 3528 for storing software and/or data.Examples of such mass storage devices 3528 include floppy disk drives,hard drive disks, compact disk drives, Blu-ray disk drives, RAIDsystems, and digital versatile disk (DVD) drives.

The coded instructions 3532 of FIG. 35 may be stored in the mass storagedevice 3528, in the volatile memory 3514, in the non-volatile memory3516, and/or on a removable tangible computer readable storage mediumsuch as a CD or DVD.

From the foregoing, it will be appreciated that the above disclosedmethods, apparatus, and articles of manufacture have been disclosed tomonitor, process, and improve operation of imaging and/or otherhealthcare systems using a plurality of deep learning and/or othermachine learning techniques.

The methods, apparatus, and articles of manufacture described above canbe applied to a variety of healthcare and non-healthcare systems. In oneparticular example, the methods, apparatus, and articles of manufacturedescribed above can be applied to the components, configuration, andoperation of a CT imaging system. FIGS. 36-38 illustrate an exampleimplementation of the imaging device 1410 as a CT imaging scanner towhich the methods, apparatus, and articles of manufacture disclosedherein can be applied. FIGS. 36 and 37 show a computed tomography (CT)imaging system 10 including a gantry 12. Gantry 12 has a rotary member13 with an x-ray source 14 that projects a beam of x-rays 16 toward adetector assembly 18 on the opposite side of the rotary member 13. Amain bearing may be utilized to attach the rotary member 13 to thestationary structure of the gantry 12. X-ray source 14 includes either astationary target or a rotating target. Detector assembly 18 is formedby a plurality of detectors 20 and data acquisition systems (DAS) 22,and can include a collimator. The plurality of detectors 20 sense theprojected x-rays that pass through a subject 24, and DAS 22 converts thedata to digital signals for subsequent processing. Each detector 20produces an analog or digital electrical signal that represents theintensity of an impinging x-ray beam and hence the attenuated beam as itpasses through subject 24. During a scan to acquire x-ray projectiondata, rotary member 13 and the components mounted thereon can rotateabout a center of rotation.

Rotation of rotary member 13 and the operation of x-ray source 14 aregoverned by a control mechanism 26 of CT system 10. Control mechanism 26can include an x-ray controller 28 and generator 30 that provides powerand timing signals to x-ray source 14 and a gantry motor controller 32that controls the rotational speed and position of rotary member 13. Animage reconstructor 34 receives sampled and digitized x-ray data fromDAS 22 and performs high speed image reconstruction. The reconstructedimage is output to a computer 36 which stores the image in a computerstorage device 38.

Computer 36 also receives commands and scanning parameters from anoperator via operator console 40 that has some form of operatorinterface, such as a keyboard, mouse, touch sensitive controller, voiceactivated controller, or any other suitable input apparatus. Display 42allows the operator to observe the reconstructed image and other datafrom computer 36. The operator supplied commands and parameters are usedby computer 36 to provide control signals and information to DAS 22,x-ray controller 28, and gantry motor controller 32. In addition,computer 36 operates a table motor controller 44 which controls amotorized table 46 to position subject 24 and gantry 12. Particularly,table 46 moves a subject 24 through a gantry opening 48, or bore, inwhole or in part. A coordinate system 50 defines a patient or Z-axis 52along which subject 24 is moved in and out of opening 48, a gantrycircumferential or X-axis 54 along which detector assembly 18 passes,and a Y-axis 56 that passes along a direction from a focal spot of x-raytube 14 to detector assembly 18.

Thus, certain examples can apply deep learning and/or other machinelearning techniques to configuration, design, and/or operation of the CTscanner 10 and its gantry 12, rotary member 13, x-ray source 14,detector assembly 18, control mechanism 26, image reconstructor 34,computer 36, operator console 40, display 42, table controller 44, table46, and/or gantry opening 48, etc. Component configuration, operation,structure can be monitored based on input, desired output, actualoutput, etc., to learn and suggest change(s) to configuration,operation, and/or structure of the scanner 10 and/or its components, forexample.

Although certain example methods, apparatus and articles of manufacturehave been described herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

What is claimed is:
 1. An apparatus comprising: a first deployed deeplearning network associated with an acquisition engine, the acquisitionengine associated with an imaging device, the first deployed deeplearning network configured to operate with the acquisition engine togenerate a configuration for the imaging device, the first deployed deeplearning network generated and deployed from a first training deeplearning network; a second deployed deep learning network associatedwith a reconstruction engine, the reconstruction engine to receiveacquired image data from the imaging device via the acquisition engineand to reconstruct an image from the acquired image data, the seconddeployed deep learning network to operate with the reconstruction enginebased on the acquired image data, the second deployed deep learningnetwork generated and deployed from a second training deep learningnetwork; a first assessment engine with a third deployed deep learningnetwork, the assessment engine to receive output from at least one ofthe acquisition engine or the reconstruction engine to assess operationof the respective at least one of the acquisition engine or thereconstruction engine and to provide feedback to the respective at leastone of the acquisition engine or the reconstruction engine.
 2. Theapparatus of claim 1 further including: a fourth deployed deep learningnetwork associated with a diagnosis engine, the diagnosis engine tofacilitate diagnosis using the reconstructed image from thereconstruction engine, the fourth deployed deep learning network tooperate with the diagnosis engine, the fourth deployed deep learningnetwork generated and deployed from a fourth training deep learningnetwork; and a second assessment engine with a fifth deployed deeplearning network, the assessment engine to receive output from at leastone of the acquisition engine, the reconstruction engine, or thediagnosis engine to assess operation of the respective at least one ofthe acquisition engine, the reconstruction engine, or the diagnosisengine and to provide feedback to the respective at least one of theacquisition engine, the reconstruction engine, or the diagnosis engine.3. The apparatus of claim 1, wherein the acquisition engine, thereconstruction engine, and the diagnosis engine exchange feedback togenerate an indication of system health.
 4. The apparatus of claim 1,wherein the reconstruction engine is to generate the reconstructed imagefor human viewing and to process the acquired image data for computeranalysis of the image data by the diagnosis engine.
 5. The apparatus ofclaim 1, wherein the first deployed deep learning network is to generatea configuration for the imaging device based on the acquisition engine,the imaging device, and a patient variable associated with the patientto be imaged.
 6. The apparatus of claim 1, wherein at least one of thefirst deployed deep learning network, the second deployed deep learningnetwork, or the third deployed deep learning network includes aconvolutional neural network.
 7. The apparatus of claim 1, wherein atleast one of the first training deep learning network, second trainingdeep learning network, or third training deep learning network isprovided with one or more features of interest in training of thecorresponding first, second, or third deployed deep learning network. 8.A method comprising: generating a configuration for the imaging devicefor image acquisition via a first deployed deep learning networkassociated with an acquisition engine associated with the imagingdevice, the first deployed deep learning network generated and deployedfrom a first training deep learning network; monitoring, using a seconddeployed deep learning network, image reconstruction by a reconstructionengine of image data acquired by the imaging device via the acquisitionengine, the second deployed deep learning network associated with thereconstruction engine and to operate with the reconstruction enginebased on the acquired image data, the second deployed deep learningnetwork generated and deployed from a second training deep learningnetwork; assessing operation of respective at least one of theacquisition engine or the reconstruction engine based on output receivedfrom the respective at least one of the acquisition engine or thereconstruction engine; and providing feedback to the respective at leastone of the acquisition engine or the reconstruction engine.
 9. Themethod of claim 8, further including: facilitating, using a thirddeployed deep learning network, diagnosis using the reconstructed imagefrom the reconstruction engine, the third deployed deep learning networkto operate with the diagnosis engine, the third deployed deep learningnetwork generated and deployed from a third training deep learningnetwork; and assessing operation of respective at least one of theacquisition engine, the reconstruction engine, or the diagnosis enginebased on output received from the respective at least one of theacquisition engine, the reconstruction engine, or the diagnosis engineto provide feedback to the respective at least one of the acquisitionengine, the reconstruction engine, or the diagnosis engine.
 10. Themethod of claim 8, further including generating an indication of systemhealth based on an exchange of feedback among the acquisition engine,the reconstruction engine, and the diagnosis engine.
 11. The method ofclaim 8, wherein the reconstruction engine is configured to generate thereconstructed image for human viewing and to process the acquired imagedata for computer analysis of the image data by the diagnosis engine.12. The method of claim 8, wherein the first deployed deep learningnetwork is to generate a configuration for the imaging device based onthe acquisition engine, the imaging device, and a patient variableassociated with the patient to be imaged.
 13. The method of claim 8,wherein at least one of the first deployed deep learning network, thesecond deployed deep learning network, or the third deployed deeplearning network includes a convolutional neural network.
 14. The methodof claim 8, wherein at least one of the first training deep learningnetwork, second training deep learning network, or third training deeplearning network is provided with one or more features of interest intraining of the corresponding first, second, or third deployed deeplearning network.
 15. A computer readable medium comprising instructionswhich, when executed, cause a machine to at least: generate aconfiguration for the imaging device for image acquisition via a firstdeployed deep learning network associated with an acquisition engineassociated with the imaging device, the first deployed deep learningnetwork generated and deployed from a first training deep learningnetwork; monitor, using a second deployed deep learning network, imagereconstruction by a reconstruction engine of image data acquired by theimaging device via the acquisition engine, the second deployed deeplearning network associated with the reconstruction engine and tooperate with the reconstruction engine based on the acquired image data,the second deployed deep learning network generated and deployed from asecond training deep learning network; assess operation of respective atleast one of the acquisition engine or the reconstruction engine, or thediagnosis engine based on output received from the respective at leastone of the acquisition engine or the reconstruction engine; and providefeedback to the respective at least one of the acquisition engine or thereconstruction engine.
 16. The computer readable medium of claim 15,wherein the instructions, when executed, further cause the machine to:facilitate, using a third deployed deep learning network, diagnosisusing the reconstructed image from the reconstruction engine, the thirddeployed deep learning network to operate with the diagnosis engine, thethird deployed deep learning network generated and deployed from a thirdtraining deep learning network; and assess operation of respective atleast one of the acquisition engine, the reconstruction engine, or thediagnosis engine based on output received from the respective at leastone of the acquisition engine, the reconstruction engine, or thediagnosis engine to provide feedback to the respective at least one ofthe acquisition engine, the reconstruction engine, or the diagnosisengine.
 17. The computer readable medium of claim 15, wherein theinstructions, when executed, further cause the machine to generate anindication of system health based on an exchange of feedback among theacquisition engine, the reconstruction engine, and the diagnosis engine.18. The computer readable medium of claim 15, wherein the reconstructionengine is configured to generate the reconstructed image for humanviewing and to process the acquired image data for computer analysis ofthe image data by the diagnosis engine.
 19. The computer readable mediumof claim 15, wherein the first deployed deep learning network is togenerate a configuration for the imaging device based on the acquisitionengine, the imaging device, and a patient variable associated with thepatient to be imaged.
 20. The computer readable medium of claim 15,wherein at least one of the first deployed deep learning network, thesecond deployed deep learning network, or the third deployed deeplearning network includes a convolutional neural network.